[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:22 dist-test-slave-rtvz krb5kdc[896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:22 dist-test-slave-rtvz krb5kdc[896](info): set up 2 sockets
Sep 24 00:49:22 dist-test-slave-rtvz krb5kdc[896](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 00:49:23.721524 888 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:49:23 dist-test-slave-rtvz krb5kdc[896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:49:23.894435 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33047
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:24.439402 909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:24.439764 909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:24.439965 909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:24.449817 909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:49:24.450001 909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:24.450120 909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:24.450255 909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:49:24.450352 909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:49:24.463333 909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:24.464852 909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:24.466368 909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:24.480170 914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:24.480674 917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:24.480257 915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:24.481792 909 server_base.cc:1034] running on GCE node
I20260924 00:49:24.482684 909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:24.484313 909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:24.485543 909 hybrid_clock.cc:648] HybridClock initialized: now 1790210964485482 us; error 53 us; skew 500 ppm
I20260924 00:49:24.485903 909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:24 dist-test-slave-rtvz krb5kdc[896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:24.489673 909 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:49:24.491876 909 webserver.cc:460] Webserver started at http://127.0.222.62:45463/ using document root <none> and password file <none>
I20260924 00:49:24.492725 909 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:24.492888 909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:24.493285 909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:24.497411 909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "b5e878c7cebd440cb427384d8bd78578"
format_stamp: "Formatted at 2026-09-24 00:49:24 on dist-test-slave-rtvz"
server_key: "c3eeece6f0079ee905c73f5106138f31"
server_key_iv: "f92e95bcde4d7edc52d840ed52d6c9b1"
server_key_version: "encryptionkey@0"
I20260924 00:49:24.498278 909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "b5e878c7cebd440cb427384d8bd78578"
format_stamp: "Formatted at 2026-09-24 00:49:24 on dist-test-slave-rtvz"
server_key: "c3eeece6f0079ee905c73f5106138f31"
server_key_iv: "f92e95bcde4d7edc52d840ed52d6c9b1"
server_key_version: "encryptionkey@0"
I20260924 00:49:24.504859 909 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 00:49:24.509730 924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:24.511188 909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:24.511365 909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "b5e878c7cebd440cb427384d8bd78578"
format_stamp: "Formatted at 2026-09-24 00:49:24 on dist-test-slave-rtvz"
server_key: "c3eeece6f0079ee905c73f5106138f31"
server_key_iv: "f92e95bcde4d7edc52d840ed52d6c9b1"
server_key_version: "encryptionkey@0"
I20260924 00:49:24.511585 909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:24.542687 909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:24.547537 909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:24.547848 909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:24.581300 909 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33047
I20260924 00:49:24.581437 976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33047 every 8 connection(s)
I20260924 00:49:24.583089 909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:49:24.586334 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 909
I20260924 00:49:24.586751 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:49:24.589640 977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:24.589731 888 external_mini_cluster.cc:1428] Setting key e9c4c6ccda2db4c32fed157b2c39a51b
I20260924 00:49:24.608162 977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578: Bootstrap starting.
I20260924 00:49:24.613834 977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:24.615660 977 log.cc:824] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:24.620247 977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578: No bootstrap required, opened a new log
I20260924 00:49:24.633122 977 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e878c7cebd440cb427384d8bd78578" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33047 } }
I20260924 00:49:24.633803 977 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:24.633970 977 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e878c7cebd440cb427384d8bd78578, State: Initialized, Role: FOLLOWER
I20260924 00:49:24.634852 977 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e878c7cebd440cb427384d8bd78578" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33047 } }
I20260924 00:49:24.635290 977 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:49:24.635512 977 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:49:24.635782 977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 00:49:24 dist-test-slave-rtvz krb5kdc[896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:24.639389 977 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e878c7cebd440cb427384d8bd78578" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33047 } }
I20260924 00:49:24.640048 977 leader_election.cc:302] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5e878c7cebd440cb427384d8bd78578; no voters:
I20260924 00:49:24.641312 977 leader_election.cc:288] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:49:24.641481 982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:24.644800 977 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:49:24.645823 982 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [term 1 LEADER]: Becoming Leader. State: Replica: b5e878c7cebd440cb427384d8bd78578, State: Running, Role: LEADER
I20260924 00:49:24.646714 982 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e878c7cebd440cb427384d8bd78578" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33047 } }
I20260924 00:49:24.652254 983 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5e878c7cebd440cb427384d8bd78578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e878c7cebd440cb427384d8bd78578" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33047 } } }
I20260924 00:49:24.652925 983 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:24.655380 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:24.594849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.0.222.62:33047)
0924 00:49:24.596373 (+ 1524us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:24.596392 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:24.598476 (+ 2084us) server_negotiation.cc:407] Connection header received
0924 00:49:24.614958 (+ 16482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:24.614986 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:24.615509 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:24.615948 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:24.620974 (+ 5026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:24.622993 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:24.626151 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:24.627109 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:24.627267 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:24.639250 (+ 11983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:24.639261 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:24.639278 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:24.639326 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:24.645048 (+ 5722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:24.646520 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:24.646532 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:24.646542 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:24.646889 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:24.647829 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:24.647835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:24.647838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:24.648841 (+ 1003us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:24.652330 (+ 3489us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:24.652762 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 00:49:24.653557 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":794,"threads_started":1}
I20260924 00:49:24.664350 984 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5e878c7cebd440cb427384d8bd78578. Latest consensus state: current_term: 1 leader_uuid: "b5e878c7cebd440cb427384d8bd78578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e878c7cebd440cb427384d8bd78578" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33047 } } }
I20260924 00:49:24.665608 984 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578 [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:24.666229 992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:49:24.674224 992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:49:24.683920 992 catalog_manager.cc:1347] Generated new cluster ID: 469c7881ce394a1cbfe01f11bfc16aa1
I20260924 00:49:24.684165 992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:49:24.720193 992 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:49:24.722319 992 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:49:24.737157 992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5e878c7cebd440cb427384d8bd78578: Generated new TSK 0
I20260924 00:49:24.738163 992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:49:24.790303 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33047
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:25.376876 1003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:25.377274 1003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:25.377492 1003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:25.387782 1003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:25.387980 1003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:25.388222 1003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:49:25.401394 1003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33047
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:25.402871 1003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:25.404431 1003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:25.421670 1008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:25.421766 1009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:25.421797 1011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:25.422993 1003 server_base.cc:1034] running on GCE node
I20260924 00:49:25.423775 1003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:25.424961 1003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:25.426235 1003 hybrid_clock.cc:648] HybridClock initialized: now 1790210965426104 us; error 75 us; skew 500 ppm
I20260924 00:49:25.426662 1003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:25 dist-test-slave-rtvz krb5kdc[896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:25.431421 1003 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:49:25.433410 1003 webserver.cc:460] Webserver started at http://127.0.222.1:38445/ using document root <none> and password file <none>
I20260924 00:49:25.434253 1003 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:25.434441 1003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:25.434821 1003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:25.438580 1003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "03762764eeaa457cae62de7af64ff27a"
format_stamp: "Formatted at 2026-09-24 00:49:25 on dist-test-slave-rtvz"
server_key: "3357bdbcf703dbe8ab83587bfb216d8a"
server_key_iv: "93d0e24830eb59256f743b4b6146c941"
server_key_version: "encryptionkey@0"
I20260924 00:49:25.439539 1003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "03762764eeaa457cae62de7af64ff27a"
format_stamp: "Formatted at 2026-09-24 00:49:25 on dist-test-slave-rtvz"
server_key: "3357bdbcf703dbe8ab83587bfb216d8a"
server_key_iv: "93d0e24830eb59256f743b4b6146c941"
server_key_version: "encryptionkey@0"
I20260924 00:49:25.445641 1003 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 00:49:25.450670 1018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:25.452369 1003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260924 00:49:25.452586 1003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "03762764eeaa457cae62de7af64ff27a"
format_stamp: "Formatted at 2026-09-24 00:49:25 on dist-test-slave-rtvz"
server_key: "3357bdbcf703dbe8ab83587bfb216d8a"
server_key_iv: "93d0e24830eb59256f743b4b6146c941"
server_key_version: "encryptionkey@0"
I20260924 00:49:25.452847 1003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:25.478163 1003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:25.484299 1003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:25.484754 1003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:25.486198 1003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:25.488260 1003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:25.488394 1003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:25.488586 1003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:25.488683 1003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:25.558553 1003 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:43565
I20260924 00:49:25.558708 1131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:43565 every 8 connection(s)
I20260924 00:49:25.560276 1003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:49:25.563113 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 1003
I20260924 00:49:25.563374 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:49:25.564371 888 external_mini_cluster.cc:1428] Setting key 197d9796dd29f1c281a97251d10b47a0
Sep 24 00:49:25 dist-test-slave-rtvz krb5kdc[896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260924 00:49:25.589798 1134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:25.563757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:32975 (local address 127.0.222.62:33047)
0924 00:49:25.564476 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:25.564483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:25.565996 (+ 1513us) server_negotiation.cc:407] Connection header received
0924 00:49:25.569715 (+ 3719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:25.569720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:25.569799 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:25.569912 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:25.572518 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:25.573562 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:25.576689 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:25.577629 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:25.577733 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:25.582623 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:25.582628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:25.582631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:25.582667 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:25.586078 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:25.586936 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:25.586942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:25.586946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:25.587069 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:25.587588 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:25.587594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:25.587598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:25.587899 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:25.588023 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:25.588892 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 00:49:25.589161 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":403,"threads_started":1}
I20260924 00:49:25.590989 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:25.564533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33047 (local address 127.0.222.1:32975)
0924 00:49:25.565823 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:25.565872 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:25.569332 (+ 3460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:25.570258 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:25.570270 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:25.571190 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:25.572263 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:25.572294 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:25.573757 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:25.573763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:25.576479 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:25.576493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:25.577912 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:25.577920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:25.578311 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:25.579255 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:25.579287 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:25.582401 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:25.586256 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:25.586263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:25.586277 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:25.586807 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:25.587192 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:25.587196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:25.587199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:25.587465 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:25.588059 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:25.588067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:25.588482 (+ 415us) client_negotiation.cc:769] Sending connection context
0924 00:49:25.588766 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 00:49:25.589445 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":637,"threads_started":1}
I20260924 00:49:25.592813 1132 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33047
I20260924 00:49:25.593185 1132 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:25.594357 1132 heartbeater.cc:511] Master 127.0.222.62:33047 requested a full tablet report, sending...
I20260924 00:49:25.597023 941 ts_manager.cc:194] Registered new tserver with Master: 03762764eeaa457cae62de7af64ff27a (127.0.222.1:43565)
I20260924 00:49:25.598958 941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:32975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:49:25.614755 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33047
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:26.188024 1138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:26.188380 1138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:26.188592 1138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:26.198544 1138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:26.198747 1138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:26.198962 1138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:49:26.211895 1138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33047
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:26.213438 1138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:26.215188 1138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:26.232539 1143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:26.232810 1146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:26.233057 1144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:26.234659 1138 server_base.cc:1034] running on GCE node
I20260924 00:49:26.235381 1138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:26.236510 1138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:26.237747 1138 hybrid_clock.cc:648] HybridClock initialized: now 1790210966237662 us; error 40 us; skew 500 ppm
I20260924 00:49:26.238173 1138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:26 dist-test-slave-rtvz krb5kdc[896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:26.242313 1138 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:49:26.244012 1138 webserver.cc:460] Webserver started at http://127.0.222.2:46657/ using document root <none> and password file <none>
I20260924 00:49:26.244930 1138 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:26.245106 1138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:26.245503 1138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:26.249096 1138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "f574bce82a984445a7d5cbc8a5d60dc2"
format_stamp: "Formatted at 2026-09-24 00:49:26 on dist-test-slave-rtvz"
server_key: "644d84d10d146794e6a4335afa96f6cf"
server_key_iv: "9e31216b89b38ee327cea372fc454b34"
server_key_version: "encryptionkey@0"
I20260924 00:49:26.250056 1138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "f574bce82a984445a7d5cbc8a5d60dc2"
format_stamp: "Formatted at 2026-09-24 00:49:26 on dist-test-slave-rtvz"
server_key: "644d84d10d146794e6a4335afa96f6cf"
server_key_iv: "9e31216b89b38ee327cea372fc454b34"
server_key_version: "encryptionkey@0"
I20260924 00:49:26.256668 1138 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260924 00:49:26.261832 1153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:26.263589 1138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 00:49:26.263815 1138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "f574bce82a984445a7d5cbc8a5d60dc2"
format_stamp: "Formatted at 2026-09-24 00:49:26 on dist-test-slave-rtvz"
server_key: "644d84d10d146794e6a4335afa96f6cf"
server_key_iv: "9e31216b89b38ee327cea372fc454b34"
server_key_version: "encryptionkey@0"
I20260924 00:49:26.264122 1138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:26.286897 1138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:26.292568 1138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:26.292980 1138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:26.294556 1138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:26.296720 1138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:26.296844 1138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:26.297056 1138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:26.297156 1138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:26.366576 1138 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:44103
I20260924 00:49:26.366711 1266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:44103 every 8 connection(s)
I20260924 00:49:26.368217 1138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:49:26.377321 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 1138
I20260924 00:49:26.377557 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:49:26.378099 888 external_mini_cluster.cc:1428] Setting key 4e67aefb273e4dbecc8e1970d0bcdce5
Sep 24 00:49:26 dist-test-slave-rtvz krb5kdc[896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:26.394047 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:26.371514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:54311 (local address 127.0.222.62:33047)
0924 00:49:26.372173 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:26.372179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:26.373617 (+ 1438us) server_negotiation.cc:407] Connection header received
0924 00:49:26.376242 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:26.376247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:26.376327 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:26.376420 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:26.378475 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:26.379358 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:26.381357 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:26.382024 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:26.382135 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:26.386652 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:26.386658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:26.386661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:26.386695 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:26.390210 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:26.391233 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:26.391238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:26.391240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:26.391455 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:26.392055 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:26.392058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:26.392061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:26.392378 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:26.392496 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:26.393496 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0924 00:49:26.393688 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:49:26.395764 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:26.372261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33047 (local address 127.0.222.2:54311)
0924 00:49:26.373425 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:26.373497 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:26.375980 (+ 2483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:26.376712 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:26.376721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:26.377478 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:26.378305 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:26.378326 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:26.379524 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:26.379528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:26.381226 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:26.381242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:26.382316 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:26.382324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:26.382615 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:26.383333 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:26.383376 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:26.386380 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:26.390352 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:26.390363 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:26.390384 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:26.391081 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:26.391580 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:26.391587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:26.391591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:26.391929 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:26.392509 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:26.392519 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:26.393037 (+ 518us) client_negotiation.cc:769] Sending connection context
0924 00:49:26.393413 (+ 376us) client_negotiation.cc:240] Negotiation successful
0924 00:49:26.394069 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":575,"threads_started":1}
I20260924 00:49:26.397322 1267 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33047
I20260924 00:49:26.397666 1267 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:49:26.398825 1267 heartbeater.cc:511] Master 127.0.222.62:33047 requested a full tablet report, sending...
I20260924 00:49:26.400669 941 ts_manager.cc:194] Registered new tserver with Master: f574bce82a984445a7d5cbc8a5d60dc2 (127.0.222.2:44103)
I20260924 00:49:26.401997 941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:54311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:49:26.420756 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33047
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:49:26.602387 1132 heartbeater.cc:503] Master 127.0.222.62:33047 was elected leader, sending a full tablet report...
W20260924 00:49:26.989866 1273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:26.990262 1273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:26.990473 1273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:27.000329 1273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:27.000540 1273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:27.000752 1273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:49:27.014346 1273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33047
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:27.015846 1273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:27.017691 1273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:27.035476 1278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:27.037400 1273 server_base.cc:1034] running on GCE node
W20260924 00:49:27.037557 1279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:27.037561 1281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:27.038750 1273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:27.040081 1273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:27.041347 1273 hybrid_clock.cc:648] HybridClock initialized: now 1790210967041245 us; error 40 us; skew 500 ppm
I20260924 00:49:27.041793 1273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:27.046371 1273 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:49:27.048278 1273 webserver.cc:460] Webserver started at http://127.0.222.3:42419/ using document root <none> and password file <none>
I20260924 00:49:27.049181 1273 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:27.049435 1273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:27.049834 1273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:27.053694 1273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "f0caf108329244cb9679c727cf013e47"
format_stamp: "Formatted at 2026-09-24 00:49:27 on dist-test-slave-rtvz"
server_key: "d23b1443c266428159bcbc02aa5870dd"
server_key_iv: "5146fd8076ff6adb892e28c4e3710d9f"
server_key_version: "encryptionkey@0"
I20260924 00:49:27.054687 1273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "f0caf108329244cb9679c727cf013e47"
format_stamp: "Formatted at 2026-09-24 00:49:27 on dist-test-slave-rtvz"
server_key: "d23b1443c266428159bcbc02aa5870dd"
server_key_iv: "5146fd8076ff6adb892e28c4e3710d9f"
server_key_version: "encryptionkey@0"
I20260924 00:49:27.061970 1273 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260924 00:49:27.067278 1288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:27.069065 1273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260924 00:49:27.069358 1273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "f0caf108329244cb9679c727cf013e47"
format_stamp: "Formatted at 2026-09-24 00:49:27 on dist-test-slave-rtvz"
server_key: "d23b1443c266428159bcbc02aa5870dd"
server_key_iv: "5146fd8076ff6adb892e28c4e3710d9f"
server_key_version: "encryptionkey@0"
I20260924 00:49:27.069643 1273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:27.086015 1273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:27.091638 1273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:27.092023 1273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:27.093506 1273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:27.095507 1273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:27.095613 1273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:27.095764 1273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:27.095857 1273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:27.167774 1273 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:35943
I20260924 00:49:27.167960 1401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:35943 every 8 connection(s)
I20260924 00:49:27.169417 1273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:49:27.173593 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 1273
I20260924 00:49:27.173866 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:49:27.174646 888 external_mini_cluster.cc:1428] Setting key f8113e69e84c68ab7396962880725af7
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:27.196352 1404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:27.172837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:57159 (local address 127.0.222.62:33047)
0924 00:49:27.173646 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:27.173651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:27.178201 (+ 4550us) server_negotiation.cc:407] Connection header received
0924 00:49:27.178260 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:27.178265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:27.178386 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:27.178497 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:27.180491 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.181545 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.183495 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.184203 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.184298 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:27.188713 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:27.188719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:27.188723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:27.188760 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:27.192829 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.193762 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.193766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.193768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.193921 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.194435 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.194439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.194441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.194803 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:27.194955 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:27.195703 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 00:49:27.195901 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":513,"threads_started":1}
I20260924 00:49:27.197527 1405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:27.173450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33047 (local address 127.0.222.3:57159)
0924 00:49:27.174807 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:27.174858 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:27.177833 (+ 2975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:27.178730 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:27.178741 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:27.179502 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:27.180321 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:27.180342 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.181711 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.181715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:27.183361 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:27.183373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.184427 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.184434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:27.184715 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:27.185457 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:27.185480 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:27.188414 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:27.192984 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:27.192994 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:27.193010 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:27.193651 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:27.194017 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:27.194021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:27.194023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:27.194290 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:27.194908 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:27.194914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:27.195307 (+ 393us) client_negotiation.cc:769] Sending connection context
0924 00:49:27.195554 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 00:49:27.196063 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"thread_start_us":712,"threads_started":1}
I20260924 00:49:27.199450 1402 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33047
I20260924 00:49:27.199788 1402 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:27.200814 1402 heartbeater.cc:511] Master 127.0.222.62:33047 requested a full tablet report, sending...
I20260924 00:49:27.202607 941 ts_manager.cc:194] Registered new tserver with Master: f0caf108329244cb9679c727cf013e47 (127.0.222.3:35943)
I20260924 00:49:27.203795 941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:57159
I20260924 00:49:27.204662 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210967, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:49:27.255450 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:27.234905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.0.222.1:43565)
0924 00:49:27.235695 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:27.235704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:27.235796 (+ 92us) server_negotiation.cc:407] Connection header received
0924 00:49:27.236035 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:27.236043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:27.236263 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:27.236428 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:27.238802 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.239833 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.241672 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.242259 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.242380 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:27.247838 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:27.247851 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:27.247868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:27.247921 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:27.252007 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.252803 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.252811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.252813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.253014 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.253519 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.253523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.253525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.253973 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:27.254338 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:27.254641 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 00:49:27.254949 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":373,"threads_started":1}
W20260924 00:49:27.257059 1046 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58776
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:49:27.303881 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:27.286255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.0.222.1:43565)
0924 00:49:27.286502 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:27.286508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:27.286783 (+ 275us) server_negotiation.cc:407] Connection header received
0924 00:49:27.287032 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:27.287037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:27.287131 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:27.287214 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:27.289344 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.290082 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.291688 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.292116 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.292211 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:27.297348 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:27.297354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:27.297358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:27.297418 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:27.300805 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.301603 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.301607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.301610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.301806 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.302303 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.302307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.302309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.302651 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:27.302976 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:27.303254 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 00:49:27.303475 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 00:49:27.306012 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 1003
I20260924 00:49:27.323851 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:27.307339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.0.222.1:43565)
0924 00:49:27.307555 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:27.307560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:27.307579 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:49:27.307693 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:27.307697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:27.307754 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:27.307844 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:27.309698 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.310607 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.312385 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.313027 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.313150 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:27.316534 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:27.316541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:27.316544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:27.316586 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:27.320494 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.321466 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.321470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.321473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.321699 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.322157 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.322161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.322163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.322502 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:27.322793 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:27.323147 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 00:49:27.323368 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:49:27.324795 1127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:27.405331 1267 heartbeater.cc:503] Master 127.0.222.62:33047 was elected leader, sending a full tablet report...
I20260924 00:49:27.413312 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 1003
I20260924 00:49:27.427184 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 1138
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:49:27.443336 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:27.428090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.0.222.2:44103)
0924 00:49:27.428713 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:27.428720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:27.428745 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:49:27.428807 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:27.428814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:27.429048 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:27.429192 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:27.430415 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.431412 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.432648 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.433287 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.433401 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:27.437672 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:27.437686 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:27.437701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:27.437739 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:27.440583 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.441266 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.441274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.441276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.441416 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.441804 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.441808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.441810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.442146 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:27.442442 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:27.442726 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 00:49:27.442998 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":330,"threads_started":1}
I20260924 00:49:27.444885 1262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:27.528270 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 1138
I20260924 00:49:27.547227 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 1273
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:49:27.564494 1419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:27.548210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.0.222.3:35943)
0924 00:49:27.548868 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:27.548875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:27.548915 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:49:27.548978 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:27.548985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:27.549265 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:27.549426 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:27.550780 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.551827 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.553105 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:27.553791 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:27.553917 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:27.558455 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:27.558470 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:27.558487 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:27.558528 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:27.561663 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.562408 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.562416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.562418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.562552 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:27.562911 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:27.562915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:27.562917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:27.563286 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:27.563530 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:27.563961 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 00:49:27.564194 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":340,"threads_started":1}
I20260924 00:49:27.565881 1397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:27.654721 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 1273
I20260924 00:49:27.668632 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 909
I20260924 00:49:27.670022 972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:27.772722 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 909
2026-09-24T00:49:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5063 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[1424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[1424](info): set up 2 sockets
Sep 24 00:49:27 dist-test-slave-rtvz krb5kdc[1424](info): commencing operation
krb5kdc: starting...
W20260924 00:49:28.784988 888 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:49:28 dist-test-slave-rtvz krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:49:28.957538 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42481
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:29.276072 1437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:29.276384 1437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:29.276518 1437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:29.283721 1437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:49:29.283838 1437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:29.283926 1437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:29.284001 1437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:49:29.284062 1437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:49:29.293958 1437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:29.295352 1437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:29.296663 1437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:29.308228 1442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:29.308444 1443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:29.308414 1445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:29.309496 1437 server_base.cc:1034] running on GCE node
I20260924 00:49:29.310292 1437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:29.311870 1437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:29.313089 1437 hybrid_clock.cc:648] HybridClock initialized: now 1790210969313027 us; error 51 us; skew 500 ppm
I20260924 00:49:29.313467 1437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:29 dist-test-slave-rtvz krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:29.316543 1437 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:49:29.318161 1437 webserver.cc:460] Webserver started at http://127.0.222.62:43381/ using document root <none> and password file <none>
I20260924 00:49:29.318872 1437 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:29.318994 1437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:29.319295 1437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:29.322470 1437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "9787374e981848499afaf1c66dc8c5b1"
format_stamp: "Formatted at 2026-09-24 00:49:29 on dist-test-slave-rtvz"
server_key: "28297cc7b083db2f8099bf2a6ae7043d"
server_key_iv: "5f076a17197095211ad1cdb147362b81"
server_key_version: "encryptionkey@0"
I20260924 00:49:29.323282 1437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "9787374e981848499afaf1c66dc8c5b1"
format_stamp: "Formatted at 2026-09-24 00:49:29 on dist-test-slave-rtvz"
server_key: "28297cc7b083db2f8099bf2a6ae7043d"
server_key_iv: "5f076a17197095211ad1cdb147362b81"
server_key_version: "encryptionkey@0"
I20260924 00:49:29.329073 1437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:49:29.332921 1452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:29.334393 1437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:49:29.334570 1437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "9787374e981848499afaf1c66dc8c5b1"
format_stamp: "Formatted at 2026-09-24 00:49:29 on dist-test-slave-rtvz"
server_key: "28297cc7b083db2f8099bf2a6ae7043d"
server_key_iv: "5f076a17197095211ad1cdb147362b81"
server_key_version: "encryptionkey@0"
I20260924 00:49:29.334762 1437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:29.362722 1437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:29.366855 1437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:29.367131 1437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:29.384925 1437 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42481
I20260924 00:49:29.384984 1503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42481 every 8 connection(s)
I20260924 00:49:29.386299 1437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:49:29.390100 1504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:29.394052 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 1437
I20260924 00:49:29.394272 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:49:29.394877 888 external_mini_cluster.cc:1428] Setting key 020356ed9aa9f105aab3950040cd2e17
I20260924 00:49:29.401602 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1: Bootstrap starting.
I20260924 00:49:29.405628 1504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:29.406883 1504 log.cc:824] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1: Log is configured to *not* fsync() on all Append() calls
Sep 24 00:49:29 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:29.410876 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1: No bootstrap required, opened a new log
I20260924 00:49:29.419838 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:29.397258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.0.222.62:42481)
0924 00:49:29.398033 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:29.398046 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:29.398088 (+ 42us) server_negotiation.cc:407] Connection header received
0924 00:49:29.399973 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:29.400014 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:29.400425 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:29.400793 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:29.402795 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:29.404205 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:29.405991 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:29.406758 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:29.406902 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:29.411625 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:29.411634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:29.411652 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:29.411693 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:29.414876 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:29.415999 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:29.416007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:29.416014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:29.416232 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:29.416715 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:29.416719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:29.416722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:29.417501 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:29.417746 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:29.418300 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 00:49:29.418755 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":343,"threads_started":1}
I20260924 00:49:29.419925 1504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9787374e981848499afaf1c66dc8c5b1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42481 } }
I20260924 00:49:29.420353 1504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:29.420456 1504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9787374e981848499afaf1c66dc8c5b1, State: Initialized, Role: FOLLOWER
I20260924 00:49:29.421015 1504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9787374e981848499afaf1c66dc8c5b1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42481 } }
I20260924 00:49:29.421280 1504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:49:29.421430 1504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:49:29.421618 1504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:29.424378 1504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9787374e981848499afaf1c66dc8c5b1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42481 } }
I20260924 00:49:29.424826 1504 leader_election.cc:302] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9787374e981848499afaf1c66dc8c5b1; no voters:
I20260924 00:49:29.425407 1504 leader_election.cc:288] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:49:29.425529 1509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:29.426203 1509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [term 1 LEADER]: Becoming Leader. State: Replica: 9787374e981848499afaf1c66dc8c5b1, State: Running, Role: LEADER
I20260924 00:49:29.426784 1509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9787374e981848499afaf1c66dc8c5b1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42481 } }
I20260924 00:49:29.427520 1504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:49:29.430292 1511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9787374e981848499afaf1c66dc8c5b1. Latest consensus state: current_term: 1 leader_uuid: "9787374e981848499afaf1c66dc8c5b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9787374e981848499afaf1c66dc8c5b1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42481 } } }
I20260924 00:49:29.430292 1510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9787374e981848499afaf1c66dc8c5b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9787374e981848499afaf1c66dc8c5b1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42481 } } }
I20260924 00:49:29.430920 1510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:29.430744 1511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1 [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:29.433508 1515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:49:29.439386 1515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:49:29.448567 1515 catalog_manager.cc:1347] Generated new cluster ID: 67346abe1c2c4c81a8a8057bfd07b489
I20260924 00:49:29.448686 1515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:49:29.477070 1515 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:49:29.478426 1515 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:49:29.486936 1515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9787374e981848499afaf1c66dc8c5b1: Generated new TSK 0
I20260924 00:49:29.488070 1515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:49:29.539412 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42481
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:29.880429 1530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:29.880746 1530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:29.880882 1530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:29.888125 1530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:29.888260 1530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:29.888422 1530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:49:29.898837 1530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:29.900182 1530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:29.901552 1530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:29.914525 1538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:29.914464 1535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:29.914562 1536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:29.915452 1530 server_base.cc:1034] running on GCE node
I20260924 00:49:29.916255 1530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:29.917326 1530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:29.918572 1530 hybrid_clock.cc:648] HybridClock initialized: now 1790210969918516 us; error 46 us; skew 500 ppm
I20260924 00:49:29.919023 1530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:29 dist-test-slave-rtvz krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:29.922297 1530 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:49:29.923823 1530 webserver.cc:460] Webserver started at http://127.0.222.1:35067/ using document root <none> and password file <none>
I20260924 00:49:29.924525 1530 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:29.924628 1530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:29.924902 1530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:29.928032 1530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "a51162351fdc48979340fa16e8390e7d"
format_stamp: "Formatted at 2026-09-24 00:49:29 on dist-test-slave-rtvz"
server_key: "0fa61f21afd1dce43d9cf9371c74e91b"
server_key_iv: "f992c202bcc2b57ffbf2af2a041b07fc"
server_key_version: "encryptionkey@0"
I20260924 00:49:29.928815 1530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "a51162351fdc48979340fa16e8390e7d"
format_stamp: "Formatted at 2026-09-24 00:49:29 on dist-test-slave-rtvz"
server_key: "0fa61f21afd1dce43d9cf9371c74e91b"
server_key_iv: "f992c202bcc2b57ffbf2af2a041b07fc"
server_key_version: "encryptionkey@0"
I20260924 00:49:29.934202 1530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 00:49:29.938045 1545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:29.939838 1530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:29.940009 1530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "a51162351fdc48979340fa16e8390e7d"
format_stamp: "Formatted at 2026-09-24 00:49:29 on dist-test-slave-rtvz"
server_key: "0fa61f21afd1dce43d9cf9371c74e91b"
server_key_iv: "f992c202bcc2b57ffbf2af2a041b07fc"
server_key_version: "encryptionkey@0"
I20260924 00:49:29.940218 1530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:29.957053 1530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:29.961465 1530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:29.961736 1530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:29.962731 1530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:29.964126 1530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:29.964201 1530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:29.964303 1530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:29.964365 1530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:29.996150 1530 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:34543
I20260924 00:49:29.996192 1659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:34543 every 8 connection(s)
I20260924 00:49:29.997534 1530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:49:29.998114 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 1530
I20260924 00:49:29.998342 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:49:29.999042 888 external_mini_cluster.cc:1428] Setting key 258c350b85fbf6ce17b6d31d365ec331
Sep 24 00:49:30 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260924 00:49:30.018822 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:30.000680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:44745 (local address 127.0.222.62:42481)
0924 00:49:30.001275 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:30.001281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:30.002398 (+ 1117us) server_negotiation.cc:407] Connection header received
0924 00:49:30.004282 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:30.004288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:30.004384 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:30.004479 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:30.006148 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.006870 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.008450 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.008862 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.008990 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:30.012943 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:30.012948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:30.012951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:30.012986 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:30.015663 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:30.016407 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:30.016411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:30.016413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:30.016537 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:30.016991 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:30.016995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:30.016997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:30.017333 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:30.017483 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:30.018206 (+ 723us) server_negotiation.cc:299] Negotiation successful
0924 00:49:30.018438 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":451,"threads_started":1}
I20260924 00:49:30.019812 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:30.001402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42481 (local address 127.0.222.1:44745)
0924 00:49:30.002237 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:30.002277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:30.004008 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:30.004753 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:30.004762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:30.005359 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:30.005970 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:30.005990 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.007037 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.007041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:30.008305 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:30.008318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.009102 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.009109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:30.009395 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:30.010155 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:30.010180 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:30.012688 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:30.015810 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:30.015820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:30.015839 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:30.016276 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:30.016643 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:30.016647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:30.016652 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:30.016874 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:30.017456 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:30.017465 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:30.017829 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 00:49:30.018076 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 00:49:30.018659 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":351,"threads_started":1}
I20260924 00:49:30.021553 1660 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42481
I20260924 00:49:30.021934 1660 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:30.022769 1660 heartbeater.cc:511] Master 127.0.222.62:42481 requested a full tablet report, sending...
I20260924 00:49:30.025115 1469 ts_manager.cc:194] Registered new tserver with Master: a51162351fdc48979340fa16e8390e7d (127.0.222.1:34543)
I20260924 00:49:30.026607 1469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:44745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:49:30.040387 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42481
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:30.362004 1666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:30.362319 1666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:30.362459 1666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:30.369846 1666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:30.369985 1666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:30.370142 1666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:49:30.380098 1666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:30.381484 1666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:30.382836 1666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:30.396064 1674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:30.396247 1672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:30.396402 1671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:30.397570 1666 server_base.cc:1034] running on GCE node
I20260924 00:49:30.398295 1666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:30.399402 1666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:30.400638 1666 hybrid_clock.cc:648] HybridClock initialized: now 1790210970400578 us; error 44 us; skew 500 ppm
I20260924 00:49:30.400931 1666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:30 dist-test-slave-rtvz krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:30.403978 1666 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:49:30.405499 1666 webserver.cc:460] Webserver started at http://127.0.222.2:43161/ using document root <none> and password file <none>
I20260924 00:49:30.406183 1666 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:30.406312 1666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:30.406584 1666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:30.409518 1666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "4ecf0626e6244e7aa778be44120e3d8e"
format_stamp: "Formatted at 2026-09-24 00:49:30 on dist-test-slave-rtvz"
server_key: "9e45ace3fd2d678cf38ad438aa785d1c"
server_key_iv: "dcc5e036f9578e08859c1760660b5b31"
server_key_version: "encryptionkey@0"
I20260924 00:49:30.410311 1666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "4ecf0626e6244e7aa778be44120e3d8e"
format_stamp: "Formatted at 2026-09-24 00:49:30 on dist-test-slave-rtvz"
server_key: "9e45ace3fd2d678cf38ad438aa785d1c"
server_key_iv: "dcc5e036f9578e08859c1760660b5b31"
server_key_version: "encryptionkey@0"
I20260924 00:49:30.415867 1666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 00:49:30.419643 1681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:30.421178 1666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 00:49:30.421380 1666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "4ecf0626e6244e7aa778be44120e3d8e"
format_stamp: "Formatted at 2026-09-24 00:49:30 on dist-test-slave-rtvz"
server_key: "9e45ace3fd2d678cf38ad438aa785d1c"
server_key_iv: "dcc5e036f9578e08859c1760660b5b31"
server_key_version: "encryptionkey@0"
I20260924 00:49:30.421581 1666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:30.433795 1666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:30.438015 1666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:30.438289 1666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:30.439282 1666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:30.440701 1666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:30.440771 1666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:30.440855 1666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:30.440900 1666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:30.471346 1666 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39291
I20260924 00:49:30.471416 1794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39291 every 8 connection(s)
I20260924 00:49:30.472621 1666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:49:30.478030 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 1666
I20260924 00:49:30.478210 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:49:30.478617 888 external_mini_cluster.cc:1428] Setting key b46f86c9d7074da6d9a0fe1280527736
Sep 24 00:49:30 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:30.492875 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:30.475493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:39827 (local address 127.0.222.62:42481)
0924 00:49:30.475673 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:30.475677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:30.476962 (+ 1285us) server_negotiation.cc:407] Connection header received
0924 00:49:30.478861 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:30.478875 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:30.478969 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:30.479065 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:30.480728 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.481499 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.483025 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.483437 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.483543 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:30.487220 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:30.487225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:30.487228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:30.487274 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:30.489777 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:30.490539 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:30.490543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:30.490546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:30.490674 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:30.491121 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:30.491125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:30.491127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:30.491407 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:30.491549 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:30.492282 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 00:49:30.492547 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:49:30.493992 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:30.475964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42481 (local address 127.0.222.2:39827)
0924 00:49:30.476707 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:30.476745 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:30.478570 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:30.479330 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:30.479342 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:30.479924 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:30.480575 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:30.480594 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.481639 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.481643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:30.482885 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:30.482902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.483651 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.483659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:30.483902 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:30.484618 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:30.484646 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:30.486963 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:30.489939 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:30.489951 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:30.489967 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:30.490418 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:30.490782 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:30.490787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:30.490791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:30.491010 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:30.491540 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:30.491548 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:30.491901 (+ 353us) client_negotiation.cc:769] Sending connection context
0924 00:49:30.492164 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 00:49:30.492728 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":288,"threads_started":1}
I20260924 00:49:30.495617 1795 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42481
I20260924 00:49:30.495924 1795 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:49:30.496683 1795 heartbeater.cc:511] Master 127.0.222.62:42481 requested a full tablet report, sending...
I20260924 00:49:30.498238 1469 ts_manager.cc:194] Registered new tserver with Master: 4ecf0626e6244e7aa778be44120e3d8e (127.0.222.2:39291)
I20260924 00:49:30.499037 1469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:39827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:49:30.516605 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42481
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:30.842983 1800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:30.843286 1800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:30.843389 1800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:30.851104 1800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:30.851239 1800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:30.851368 1800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:49:30.861454 1800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:30.862736 1800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:30.864099 1800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:30.878501 1805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:30.878592 1808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:30.878616 1806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:30.879567 1800 server_base.cc:1034] running on GCE node
I20260924 00:49:30.880240 1800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:30.881363 1800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:30.882625 1800 hybrid_clock.cc:648] HybridClock initialized: now 1790210970882549 us; error 73 us; skew 500 ppm
I20260924 00:49:30.882939 1800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:30 dist-test-slave-rtvz krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:30.886283 1800 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:49:30.887998 1800 webserver.cc:460] Webserver started at http://127.0.222.3:35175/ using document root <none> and password file <none>
I20260924 00:49:30.888742 1800 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:30.888866 1800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:30.889187 1800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:30.892377 1800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "5fe926b899524356b8d0d9b76d915696"
format_stamp: "Formatted at 2026-09-24 00:49:30 on dist-test-slave-rtvz"
server_key: "2c14d113d870302d32318f65b355f100"
server_key_iv: "4a55cdcf0c06e0671b0dfd0ef38eaae1"
server_key_version: "encryptionkey@0"
I20260924 00:49:30.893221 1800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "5fe926b899524356b8d0d9b76d915696"
format_stamp: "Formatted at 2026-09-24 00:49:30 on dist-test-slave-rtvz"
server_key: "2c14d113d870302d32318f65b355f100"
server_key_iv: "4a55cdcf0c06e0671b0dfd0ef38eaae1"
server_key_version: "encryptionkey@0"
I20260924 00:49:30.898662 1800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 00:49:30.902398 1815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:30.903829 1800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:49:30.904062 1800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "5fe926b899524356b8d0d9b76d915696"
format_stamp: "Formatted at 2026-09-24 00:49:30 on dist-test-slave-rtvz"
server_key: "2c14d113d870302d32318f65b355f100"
server_key_iv: "4a55cdcf0c06e0671b0dfd0ef38eaae1"
server_key_version: "encryptionkey@0"
I20260924 00:49:30.904251 1800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:30.927256 1800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:30.931775 1800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:30.932058 1800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:30.933245 1800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:30.934772 1800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:30.934859 1800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:30.934986 1800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:30.935038 1800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:30.967490 1800 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:37245
I20260924 00:49:30.967561 1929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:37245 every 8 connection(s)
I20260924 00:49:30.968791 1800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:49:30.973485 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 1800
I20260924 00:49:30.973696 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:49:30.974249 888 external_mini_cluster.cc:1428] Setting key 063efb39f25a1a07181ba54f997fdb2a
Sep 24 00:49:30 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:30.989492 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:30.971456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:51821 (local address 127.0.222.62:42481)
0924 00:49:30.971627 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:30.971631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:30.972924 (+ 1293us) server_negotiation.cc:407] Connection header received
0924 00:49:30.975463 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:30.975469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:30.975581 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:30.975668 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:30.977283 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.978042 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.979637 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.980054 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.980163 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:30.983610 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:30.983615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:30.983619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:30.983653 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:30.986148 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:30.987051 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:30.987055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:30.987058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:30.987192 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:30.987623 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:30.987627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:30.987629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:30.987916 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:30.988067 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:30.988826 (+ 759us) server_negotiation.cc:299] Negotiation successful
0924 00:49:30.989074 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:49:30.990607 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:30.971934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42481 (local address 127.0.222.3:51821)
0924 00:49:30.972734 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:30.972779 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:30.975160 (+ 2381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:30.975940 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:30.975950 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:30.976501 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:30.977103 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:30.977121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.978188 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.978193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:30.979488 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:30.979501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:30.980286 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:30.980293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:30.980537 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:30.981190 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:30.981257 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:30.983376 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:30.986296 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:30.986306 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:30.986325 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:30.986919 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:30.987295 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:30.987300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:30.987304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:30.987508 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:30.988052 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:30.988060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:30.988424 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 00:49:30.988691 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 00:49:30.989383 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":331,"threads_started":1}
I20260924 00:49:30.992069 1930 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42481
I20260924 00:49:30.992367 1930 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:30.993129 1930 heartbeater.cc:511] Master 127.0.222.62:42481 requested a full tablet report, sending...
I20260924 00:49:30.994786 1469 ts_manager.cc:194] Registered new tserver with Master: 5fe926b899524356b8d0d9b76d915696 (127.0.222.3:37245)
I20260924 00:49:30.995795 1469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:51821
I20260924 00:49:31.002616 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:49:31.025303 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.009144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.222.62:42481)
0924 00:49:31.009358 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.009363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.009863 (+ 500us) server_negotiation.cc:407] Connection header received
0924 00:49:31.010076 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.010082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.010205 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.010325 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:31.012307 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.013119 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.015010 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.015403 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.015516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.018688 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:31.018694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:31.018699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:31.018733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:31.022145 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.022953 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.022957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.022960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.023155 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.023570 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.023574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.023576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.024069 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:31.024355 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.024648 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.024821 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:49:31.029898 1660 heartbeater.cc:503] Master 127.0.222.62:42481 was elected leader, sending a full tablet report...
I20260924 00:49:31.033077 1469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:49:31.043716 1469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:49:31.087371 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.060370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34942 (local address 127.0.222.2:39291)
0924 00:49:31.061253 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.061265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.075596 (+ 14331us) server_negotiation.cc:407] Connection header received
0924 00:49:31.075735 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.075745 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.076091 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.076618 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:31.077828 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.079710 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.083518 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.085777 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.085900 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.086143 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.086718 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.086945 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":1322,"threads_started":1}
I20260924 00:49:31.087641 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.060062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:39291 (local address 127.0.0.1:34942)
0924 00:49:31.074716 (+ 14654us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:31.074776 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:31.075130 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:31.076400 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:31.076413 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:31.076447 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:31.077528 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.077547 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.079852 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.079858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.083282 (+ 3424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.083323 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.086283 (+ 2960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.086288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.086465 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.086503 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:49:31.086629 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 00:49:31.086831 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14236,"mutex_wait_us":23,"spinlock_wait_cycles":64384,"thread_start_us":13892,"threads_started":1}
I20260924 00:49:31.088763 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.062423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34543 (local address 127.0.0.1:43384)
0924 00:49:31.076677 (+ 14254us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:31.076697 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:31.077295 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:31.078019 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:31.078030 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:31.078061 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:31.079184 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.079202 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.081562 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.081567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.084334 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.084350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.087541 (+ 3191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.087549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.087791 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.087841 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 00:49:31.087955 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 00:49:31.088100 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14063,"mutex_wait_us":57,"thread_start_us":13791,"threads_started":1}
I20260924 00:49:31.089030 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.061160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.0.222.3:37245)
0924 00:49:31.071730 (+ 10570us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.071743 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.076389 (+ 4646us) server_negotiation.cc:407] Connection header received
0924 00:49:31.076740 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.076749 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.077067 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.077313 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:31.079150 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.081715 (+ 2565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.085035 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.087047 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.087184 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.087497 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.088212 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.088431 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10258,"thread_start_us":12011,"threads_started":1}
I20260924 00:49:31.089898 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.063058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.0.222.1:34543)
0924 00:49:31.075759 (+ 12701us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.075768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.077022 (+ 1254us) server_negotiation.cc:407] Connection header received
0924 00:49:31.077511 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.077518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.077777 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.078003 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:31.079385 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.081370 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.084540 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.086764 (+ 2224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.086928 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.087259 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.089115 (+ 1856us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.089419 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12467,"thread_start_us":12232,"threads_started":1}
I20260924 00:49:31.089968 1944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.061296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:37245 (local address 127.0.0.1:51604)
0924 00:49:31.076238 (+ 14942us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:31.076265 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:31.076530 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:31.077729 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:31.077734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:31.077758 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:31.078850 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.078868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.081881 (+ 3013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.081887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.084778 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.084794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.087724 (+ 2930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.087732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.087945 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.087979 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 00:49:31.088058 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 00:49:31.089367 (+ 1309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14737,"mutex_wait_us":39,"thread_start_us":14413,"threads_started":1}
I20260924 00:49:31.092257 1730 tablet_service.cc:1467] Processing CreateTablet for tablet 55634671162742c1bbb842363a21eceb (DEFAULT_TABLE table=test-table [id=c0daa6b8b7904198aebf1b2c12ab014b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:31.094158 1730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55634671162742c1bbb842363a21eceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:31.094067 1864 tablet_service.cc:1467] Processing CreateTablet for tablet 55634671162742c1bbb842363a21eceb (DEFAULT_TABLE table=test-table [id=c0daa6b8b7904198aebf1b2c12ab014b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:31.094462 1594 tablet_service.cc:1467] Processing CreateTablet for tablet 55634671162742c1bbb842363a21eceb (DEFAULT_TABLE table=test-table [id=c0daa6b8b7904198aebf1b2c12ab014b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:31.095602 1864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55634671162742c1bbb842363a21eceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:31.096016 1594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55634671162742c1bbb842363a21eceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:31.112103 1948 tablet_bootstrap.cc:492] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696: Bootstrap starting.
I20260924 00:49:31.114925 1949 tablet_bootstrap.cc:492] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e: Bootstrap starting.
I20260924 00:49:31.116112 1950 tablet_bootstrap.cc:492] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d: Bootstrap starting.
I20260924 00:49:31.120188 1948 tablet_bootstrap.cc:654] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:31.120476 1949 tablet_bootstrap.cc:654] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:31.120754 1950 tablet_bootstrap.cc:654] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:31.122359 1950 log.cc:824] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:31.122473 1949 log.cc:824] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:31.122491 1948 log.cc:824] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:31.126703 1950 tablet_bootstrap.cc:492] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d: No bootstrap required, opened a new log
I20260924 00:49:31.127074 1950 ts_tablet_manager.cc:1397] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260924 00:49:31.128100 1949 tablet_bootstrap.cc:492] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e: No bootstrap required, opened a new log
I20260924 00:49:31.128497 1949 ts_tablet_manager.cc:1397] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260924 00:49:31.128779 1948 tablet_bootstrap.cc:492] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696: No bootstrap required, opened a new log
I20260924 00:49:31.129257 1948 ts_tablet_manager.cc:1397] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20260924 00:49:31.139097 1950 raft_consensus.cc:348] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.139670 1950 raft_consensus.cc:374] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:31.139822 1950 raft_consensus.cc:729] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51162351fdc48979340fa16e8390e7d, State: Initialized, Role: FOLLOWER
I20260924 00:49:31.140605 1950 consensus_queue.cc:260] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.143821 1950 ts_tablet_manager.cc:1428] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260924 00:49:31.143993 1949 raft_consensus.cc:348] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.144742 1949 raft_consensus.cc:374] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:31.144940 1949 raft_consensus.cc:729] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecf0626e6244e7aa778be44120e3d8e, State: Initialized, Role: FOLLOWER
I20260924 00:49:31.145768 1949 consensus_queue.cc:260] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.145663 1948 raft_consensus.cc:348] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.146247 1948 raft_consensus.cc:374] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:31.146404 1948 raft_consensus.cc:729] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe926b899524356b8d0d9b76d915696, State: Initialized, Role: FOLLOWER
I20260924 00:49:31.147540 1795 heartbeater.cc:503] Master 127.0.222.62:42481 was elected leader, sending a full tablet report...
I20260924 00:49:31.147423 1948 consensus_queue.cc:260] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.148468 1949 ts_tablet_manager.cc:1428] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
I20260924 00:49:31.150300 1955 raft_consensus.cc:482] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:31.150684 1955 raft_consensus.cc:504] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.156032 1955 leader_election.cc:288] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fe926b899524356b8d0d9b76d915696 (127.0.222.3:37245), a51162351fdc48979340fa16e8390e7d (127.0.222.1:34543)
I20260924 00:49:31.159675 1930 heartbeater.cc:503] Master 127.0.222.62:42481 was elected leader, sending a full tablet report...
I20260924 00:49:31.163180 1948 ts_tablet_manager.cc:1428] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696: Time spent starting tablet: real 0.034s user 0.006s sys 0.018s
I20260924 00:49:31.165467 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.153844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:42347 (local address 127.0.222.1:34543)
0924 00:49:31.154064 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.154069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.156228 (+ 2159us) server_negotiation.cc:407] Connection header received
0924 00:49:31.156538 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.156545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.156628 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.156695 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:31.158134 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.159551 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.162591 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.164148 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.164305 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.164450 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.164904 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.165087 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:49:31.165668 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.153320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34543 (local address 127.0.222.2:42347)
0924 00:49:31.156121 (+ 2801us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:31.156141 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:31.156355 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:31.156937 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:31.156941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:31.157051 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:31.157929 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.157944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.159767 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.159772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.162409 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.162423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.164482 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.164491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.164707 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.164743 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 00:49:31.164828 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:49:31.164983 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2649,"mutex_wait_us":47,"spinlock_wait_cycles":90880,"thread_start_us":2389,"threads_started":1}
I20260924 00:49:31.167335 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.153810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:52643 (local address 127.0.222.3:37245)
0924 00:49:31.154078 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.154084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.155382 (+ 1298us) server_negotiation.cc:407] Connection header received
0924 00:49:31.155732 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.155739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.155825 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.155888 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:31.157997 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.159399 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.164394 (+ 4995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.166010 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.166103 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.166204 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.166977 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.167080 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 00:49:31.167106 1614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55634671162742c1bbb842363a21eceb" candidate_uuid: "4ecf0626e6244e7aa778be44120e3d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51162351fdc48979340fa16e8390e7d" is_pre_election: true
I20260924 00:49:31.167606 1959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.153106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:37245 (local address 127.0.222.2:52643)
0924 00:49:31.155252 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:31.155275 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:31.155546 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:31.156442 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:31.156447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:31.156475 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:31.157766 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.157781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.160090 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.160096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.164171 (+ 4075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:31.164185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.166454 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.166462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:31.166720 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.166758 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:49:31.166851 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 00:49:31.166987 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1917,"mutex_wait_us":5,"thread_start_us":1661,"threads_started":1}
I20260924 00:49:31.168016 1614 raft_consensus.cc:2454] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecf0626e6244e7aa778be44120e3d8e in term 0.
I20260924 00:49:31.169023 1685 leader_election.cc:302] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecf0626e6244e7aa778be44120e3d8e, a51162351fdc48979340fa16e8390e7d; no voters:
I20260924 00:49:31.169008 1884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55634671162742c1bbb842363a21eceb" candidate_uuid: "4ecf0626e6244e7aa778be44120e3d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe926b899524356b8d0d9b76d915696" is_pre_election: true
I20260924 00:49:31.169541 1955 raft_consensus.cc:2781] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:31.169764 1955 raft_consensus.cc:482] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:31.169945 1955 raft_consensus.cc:3037] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:31.169812 1884 raft_consensus.cc:2454] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecf0626e6244e7aa778be44120e3d8e in term 0.
I20260924 00:49:31.173254 1955 raft_consensus.cc:504] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.174069 1955 leader_election.cc:288] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [CANDIDATE]: Term 1 election: Requested vote from peers 5fe926b899524356b8d0d9b76d915696 (127.0.222.3:37245), a51162351fdc48979340fa16e8390e7d (127.0.222.1:34543)
I20260924 00:49:31.174544 1884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55634671162742c1bbb842363a21eceb" candidate_uuid: "4ecf0626e6244e7aa778be44120e3d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe926b899524356b8d0d9b76d915696"
I20260924 00:49:31.174681 1614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55634671162742c1bbb842363a21eceb" candidate_uuid: "4ecf0626e6244e7aa778be44120e3d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51162351fdc48979340fa16e8390e7d"
I20260924 00:49:31.174772 1884 raft_consensus.cc:3037] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:31.174998 1614 raft_consensus.cc:3037] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:31.177928 1884 raft_consensus.cc:2454] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecf0626e6244e7aa778be44120e3d8e in term 1.
I20260924 00:49:31.178118 1614 raft_consensus.cc:2454] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecf0626e6244e7aa778be44120e3d8e in term 1.
I20260924 00:49:31.178376 1683 leader_election.cc:302] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecf0626e6244e7aa778be44120e3d8e, 5fe926b899524356b8d0d9b76d915696; no voters:
I20260924 00:49:31.178794 1955 raft_consensus.cc:2781] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:31.178994 1955 raft_consensus.cc:686] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [term 1 LEADER]: Becoming Leader. State: Replica: 4ecf0626e6244e7aa778be44120e3d8e, State: Running, Role: LEADER
I20260924 00:49:31.179672 1955 consensus_queue.cc:237] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } }
I20260924 00:49:31.185343 1468 catalog_manager.cc:5495] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecf0626e6244e7aa778be44120e3d8e (127.0.222.2). New cstate: current_term: 1 leader_uuid: "4ecf0626e6244e7aa778be44120e3d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecf0626e6244e7aa778be44120e3d8e" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fe926b899524356b8d0d9b76d915696" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a51162351fdc48979340fa16e8390e7d" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34543 } health_report { overall_health: UNKNOWN } } }
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
W20260924 00:49:31.220515 1931 tablet.cc:2369] T 55634671162742c1bbb842363a21eceb P 5fe926b899524356b8d0d9b76d915696: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:49:31.224990 1796 tablet.cc:2369] T 55634671162742c1bbb842363a21eceb P 4ecf0626e6244e7aa778be44120e3d8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:49:31.229497 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.207767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.0.222.1:34543)
0924 00:49:31.207993 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.207998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.209049 (+ 1051us) server_negotiation.cc:407] Connection header received
0924 00:49:31.209349 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.209355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.209511 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.209618 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:31.212406 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.213490 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.215270 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.215939 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.216068 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.221982 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:31.221999 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:31.222014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:31.222053 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:31.226127 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.226903 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.226914 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.226918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.227106 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.227548 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.227551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.227553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.228079 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:31.228432 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.228755 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.228974 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 00:49:31.250283 1661 tablet.cc:2369] T 55634671162742c1bbb842363a21eceb P a51162351fdc48979340fa16e8390e7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210971, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210971, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:49:31.280409 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.261408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.222.1:34543)
0924 00:49:31.261644 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.261651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.262104 (+ 453us) server_negotiation.cc:407] Connection header received
0924 00:49:31.262312 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.262318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.262387 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.262481 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:31.264645 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.265773 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.267457 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.268062 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.268186 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.273301 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:31.273309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:31.273314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:31.273363 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:31.277258 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.278004 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.278010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.278013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.278217 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.278736 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.278742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.278746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.279103 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:31.279428 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.279687 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.279955 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:49:31.329526 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.309800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.0.222.1:34543)
0924 00:49:31.310097 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.310104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.310638 (+ 534us) server_negotiation.cc:407] Connection header received
0924 00:49:31.310778 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.310782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.310857 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.310947 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:31.313307 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.314213 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.315969 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.316666 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.316777 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.322386 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:31.322393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:31.322396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:31.322440 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:31.326126 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.326971 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.326977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.326980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.327149 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.327591 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.327595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.327597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.328030 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:31.328355 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.328705 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.329026 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 00:49:31.331561 1574 tablet_service.cc:2908] Rejecting scan request for tablet 55634671162742c1bbb842363a21eceb: Uninitialized: clean time has not yet been initialized
I20260924 00:49:31.334947 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 1530
I20260924 00:49:31.348574 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.335826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.0.222.1:34543)
0924 00:49:31.335992 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.335997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.336361 (+ 364us) server_negotiation.cc:407] Connection header received
0924 00:49:31.336523 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.336527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.336600 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.336672 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:31.337907 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.338761 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.340058 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.340488 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.340588 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.343366 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:31.343371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:31.343374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:31.343440 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:31.345914 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.346666 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.346669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.346672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.346789 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.347213 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.347216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.347218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.347500 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:31.347713 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.348083 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.348277 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:49:31.349328 1655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:31.440404 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 1530
I20260924 00:49:31.454320 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 1666
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:49:31.469717 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.455045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34950 (local address 127.0.222.2:39291)
0924 00:49:31.455249 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.455254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.455273 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:49:31.455404 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.455409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.455489 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.455582 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:31.457168 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.457926 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.459601 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.460009 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.460117 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.464132 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:31.464147 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:31.464163 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:31.464200 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:31.466978 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.467657 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.467665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.467667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.467819 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.468169 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.468173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.468175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.468511 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:31.468803 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.469071 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.469440 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:49:31.470613 1790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:31.560286 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 1666
I20260924 00:49:31.573449 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 1800
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:49:31.588809 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:31.574443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.222.3:37245)
0924 00:49:31.574605 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:31.574610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:31.574626 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:49:31.574690 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:31.574694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:31.574773 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:31.574855 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:31.576173 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.577078 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.578514 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:31.578991 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:31.579099 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:31.583212 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:31.583229 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:31.583246 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:31.583286 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:31.586232 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.586936 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.586944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.586946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.587085 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:31.587429 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:31.587432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:31.587434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:31.587788 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:31.588120 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:31.588329 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 00:49:31.588545 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:49:31.589529 1924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:31.676858 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 1800
I20260924 00:49:31.690220 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 1437
I20260924 00:49:31.691442 1499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:31.799825 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 1437
2026-09-24T00:49:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4027 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1978](info): set up 2 sockets
Sep 24 00:49:31 dist-test-slave-rtvz krb5kdc[1978](info): commencing operation
krb5kdc: starting...
W20260924 00:49:32.789153 888 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:49:32 dist-test-slave-rtvz krb5kdc[1978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:49:32.969769 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34565
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:34565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:33.293761 1991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:33.294080 1991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:33.294200 1991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:33.301926 1991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:49:33.302055 1991 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:49:33.302106 1991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:33.302196 1991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:33.302261 1991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:49:33.302353 1991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:49:33.312613 1991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:34565
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:33.313997 1991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:33.315361 1991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:33.327713 1996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:33.327684 1999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:33.327775 1997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:33.330976 1991 server_base.cc:1034] running on GCE node
I20260924 00:49:33.331844 1991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:33.333568 1991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:33.334834 1991 hybrid_clock.cc:648] HybridClock initialized: now 1790210973334749 us; error 48 us; skew 500 ppm
I20260924 00:49:33.335213 1991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:33 dist-test-slave-rtvz krb5kdc[1978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:33.338613 1991 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:49:33.340279 1991 webserver.cc:460] Webserver started at http://127.0.222.62:35695/ using document root <none> and password file <none>
I20260924 00:49:33.341012 1991 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:33.341104 1991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:33.341470 1991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:33.344499 1991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "f7b2bbbb4f174595b4eda491a1f0942d"
format_stamp: "Formatted at 2026-09-24 00:49:33 on dist-test-slave-rtvz"
server_key: "679e2f63d18ef87cbeac2ef8f3a057e6"
server_key_iv: "4daf9ba3be84a33e6b8a91bce9ac6fac"
server_key_version: "encryptionkey@0"
I20260924 00:49:33.345340 1991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "f7b2bbbb4f174595b4eda491a1f0942d"
format_stamp: "Formatted at 2026-09-24 00:49:33 on dist-test-slave-rtvz"
server_key: "679e2f63d18ef87cbeac2ef8f3a057e6"
server_key_iv: "4daf9ba3be84a33e6b8a91bce9ac6fac"
server_key_version: "encryptionkey@0"
I20260924 00:49:33.350620 1991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 00:49:33.354188 2006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:33.355834 1991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:33.355997 1991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "f7b2bbbb4f174595b4eda491a1f0942d"
format_stamp: "Formatted at 2026-09-24 00:49:33 on dist-test-slave-rtvz"
server_key: "679e2f63d18ef87cbeac2ef8f3a057e6"
server_key_iv: "4daf9ba3be84a33e6b8a91bce9ac6fac"
server_key_version: "encryptionkey@0"
I20260924 00:49:33.356228 1991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:33.377362 1991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:33.381743 1991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:33.382004 1991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:33.402621 1991 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:34565
I20260924 00:49:33.402642 2068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:34565 every 8 connection(s)
I20260924 00:49:33.403976 1991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:49:33.406697 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 1991
I20260924 00:49:33.406908 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:49:33.407537 888 external_mini_cluster.cc:1428] Setting key 4db40549fba4d256948604d2d98a7dcc
I20260924 00:49:33.408042 2069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:33.422762 2069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d: Bootstrap starting.
Sep 24 00:49:33 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:33.426388 2069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:33.427431 2069 log.cc:824] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:33.430510 2069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d: No bootstrap required, opened a new log
I20260924 00:49:33.431866 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:33.409547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.0.222.62:34565)
0924 00:49:33.410354 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:33.410367 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:33.410409 (+ 42us) server_negotiation.cc:407] Connection header received
0924 00:49:33.412846 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:33.412873 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:33.413385 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:33.413767 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:33.415630 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.417030 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.418752 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.419547 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.419713 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:33.424513 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:33.424521 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:33.424536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:33.424574 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:33.427598 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:33.428533 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:33.428541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:33.428550 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:33.428715 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:33.429252 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:33.429256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:33.429258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:33.429777 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:33.430010 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:33.430456 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 00:49:33.430846 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":378,"threads_started":1}
I20260924 00:49:33.437916 2069 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b2bbbb4f174595b4eda491a1f0942d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34565 } }
I20260924 00:49:33.438261 2069 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:33.438369 2069 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b2bbbb4f174595b4eda491a1f0942d, State: Initialized, Role: FOLLOWER
I20260924 00:49:33.438922 2069 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b2bbbb4f174595b4eda491a1f0942d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34565 } }
I20260924 00:49:33.439136 2069 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:49:33.439256 2069 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:49:33.439460 2069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:33.441613 2069 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b2bbbb4f174595b4eda491a1f0942d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34565 } }
I20260924 00:49:33.442167 2069 leader_election.cc:302] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7b2bbbb4f174595b4eda491a1f0942d; no voters:
I20260924 00:49:33.442737 2069 leader_election.cc:288] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:49:33.442855 2074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:33.443511 2074 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [term 1 LEADER]: Becoming Leader. State: Replica: f7b2bbbb4f174595b4eda491a1f0942d, State: Running, Role: LEADER
I20260924 00:49:33.444067 2074 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b2bbbb4f174595b4eda491a1f0942d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34565 } }
I20260924 00:49:33.444685 2069 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:49:33.446864 2075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7b2bbbb4f174595b4eda491a1f0942d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b2bbbb4f174595b4eda491a1f0942d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34565 } } }
I20260924 00:49:33.447222 2075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:33.447686 2076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b2bbbb4f174595b4eda491a1f0942d. Latest consensus state: current_term: 1 leader_uuid: "f7b2bbbb4f174595b4eda491a1f0942d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b2bbbb4f174595b4eda491a1f0942d" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34565 } } }
I20260924 00:49:33.447971 2076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:33.448992 2082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:49:33.454823 2082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:49:33.464006 2082 catalog_manager.cc:1347] Generated new cluster ID: d0e20a9b814248c5875d6875d0b90a0d
I20260924 00:49:33.464141 2082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:49:33.483955 2082 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:49:33.486022 2082 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:49:33.495160 2082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d: Generated new TSK 0
I20260924 00:49:33.496243 2082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:49:33.541136 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34565
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:33.867643 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:33.867961 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:33.868103 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:33.875530 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:33.875674 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:33.875797 2095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:49:33.875913 2095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:49:33.886130 2095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:34565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:33.887501 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:33.888898 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:33.902158 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:33.902238 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:33.902416 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:33.903322 2095 server_base.cc:1034] running on GCE node
I20260924 00:49:33.903990 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:33.904942 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:33.906168 2095 hybrid_clock.cc:648] HybridClock initialized: now 1790210973906079 us; error 63 us; skew 500 ppm
I20260924 00:49:33.906474 2095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:33 dist-test-slave-rtvz krb5kdc[1978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:33.909729 2095 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:49:33.911334 2095 webserver.cc:460] Webserver started at http://127.0.222.1:38601/ using document root <none> and password file <none>
I20260924 00:49:33.912082 2095 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:33.912191 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:33.912519 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:33.915655 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "47910f31b899440ca67a3224006b22b4"
format_stamp: "Formatted at 2026-09-24 00:49:33 on dist-test-slave-rtvz"
server_key: "2fedbfdb384d0ce733e5d0753047660a"
server_key_iv: "c372097532dd2b4e2c2fdc56e1aea876"
server_key_version: "encryptionkey@0"
I20260924 00:49:33.916492 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "47910f31b899440ca67a3224006b22b4"
format_stamp: "Formatted at 2026-09-24 00:49:33 on dist-test-slave-rtvz"
server_key: "2fedbfdb384d0ce733e5d0753047660a"
server_key_iv: "c372097532dd2b4e2c2fdc56e1aea876"
server_key_version: "encryptionkey@0"
I20260924 00:49:33.922057 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 00:49:33.926111 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:33.927623 2095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:33.927793 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "47910f31b899440ca67a3224006b22b4"
format_stamp: "Formatted at 2026-09-24 00:49:33 on dist-test-slave-rtvz"
server_key: "2fedbfdb384d0ce733e5d0753047660a"
server_key_iv: "c372097532dd2b4e2c2fdc56e1aea876"
server_key_version: "encryptionkey@0"
I20260924 00:49:33.928014 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:33.948206 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:33.953338 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:33.953625 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:33.956564 2095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:33.956671 2095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:33.956828 2095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:33.956899 2095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:49:33 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:33.984841 2121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:33.958192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34565 (local address 127.0.222.1:37853)
0924 00:49:33.959079 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:33.959131 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:33.961719 (+ 2588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:33.962998 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:33.963012 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:33.964231 (+ 1219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:33.965542 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:33.965585 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.967055 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.967062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:33.969812 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:33.969837 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.970854 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.970861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:33.971218 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:33.972058 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:33.972095 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:33.975605 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:33.979783 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:33.979795 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:33.979818 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:33.980496 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:33.980881 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:33.980886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:33.980891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:33.981109 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:33.981687 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:33.981694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:33.982217 (+ 523us) client_negotiation.cc:769] Sending connection context
0924 00:49:33.982579 (+ 362us) client_negotiation.cc:240] Negotiation successful
0924 00:49:33.983207 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":368,"threads_started":1}
I20260924 00:49:33.985878 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:33.957501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:37853 (local address 127.0.222.62:34565)
0924 00:49:33.958018 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:33.958023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:33.959269 (+ 1246us) server_negotiation.cc:407] Connection header received
0924 00:49:33.962236 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:33.962241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:33.962337 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:33.962449 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:33.965842 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.966816 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.970042 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.970577 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.970722 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:33.975904 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:33.975909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:33.975913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:33.976013 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:33.979578 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:33.980634 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:33.980640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:33.980644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:33.980772 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:33.981271 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:33.981275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:33.981277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:33.981562 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:33.981712 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:33.985433 (+ 3721us) server_negotiation.cc:299] Negotiation successful
0924 00:49:33.985622 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
I20260924 00:49:34.000530 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:33871
I20260924 00:49:34.000548 2232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:33871 every 8 connection(s)
I20260924 00:49:34.001894 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:49:34.004196 2233 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34565
I20260924 00:49:34.004459 2233 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:34.005373 2233 heartbeater.cc:511] Master 127.0.222.62:34565 requested a full tablet report, sending...
I20260924 00:49:34.005769 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:33.992561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34565 (local address 127.0.222.1:39609)
0924 00:49:33.993747 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:33.993769 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:33.993951 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:33.994311 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:33.994316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:33.994776 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:33.995082 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:33.995092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.996202 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.996208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:33.997315 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:33.997328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.998268 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.998276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:33.998433 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:33.999055 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:33.999073 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:33.999993 (+ 920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:34.003031 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:34.003035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:34.003039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:34.003521 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:34.003930 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:34.003935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:34.003938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:34.004089 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:34.004707 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:34.004710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:34.004880 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 00:49:34.005023 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 00:49:34.005333 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":844,"threads_started":1}
I20260924 00:49:34.005796 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:33.993085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:39609 (local address 127.0.222.62:34565)
0924 00:49:33.993317 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:33.993324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:33.993883 (+ 559us) server_negotiation.cc:407] Connection header received
0924 00:49:33.994083 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:33.994088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:33.994156 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:33.994262 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:33.995255 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.996075 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.997457 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:33.998035 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:33.998149 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:34.000180 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:34.000184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:34.000188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:34.000218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:34.002858 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:34.003647 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:34.003651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:34.003654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:34.003808 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:34.004201 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:34.004206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:34.004209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:34.004603 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:34.004753 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:34.005232 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 00:49:34.005491 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 00:49:34.007813 2022 ts_manager.cc:194] Registered new tserver with Master: 47910f31b899440ca67a3224006b22b4 (127.0.222.1:33871)
I20260924 00:49:34.008908 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 2095
I20260924 00:49:34.009132 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:49:34.009516 2022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:37853
I20260924 00:49:34.009893 888 external_mini_cluster.cc:1428] Setting key 05c795f1126726cd19cffa5f1a6d4c20
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:49:34.049017 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34565
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:34.376801 2237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:34.377092 2237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:34.377187 2237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:34.384575 2237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:34.384688 2237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:34.384783 2237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:49:34.384868 2237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:49:34.394761 2237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:34565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:34.395998 2237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:34.397275 2237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:34.409690 2242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:34.409796 2245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:34.409948 2243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:34.411494 2237 server_base.cc:1034] running on GCE node
I20260924 00:49:34.412142 2237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:34.413064 2237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:34.414302 2237 hybrid_clock.cc:648] HybridClock initialized: now 1790210974414213 us; error 63 us; skew 500 ppm
I20260924 00:49:34.414621 2237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:34 dist-test-slave-rtvz krb5kdc[1978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:34.418047 2237 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:49:34.419524 2237 webserver.cc:460] Webserver started at http://127.0.222.2:44457/ using document root <none> and password file <none>
I20260924 00:49:34.420183 2237 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:34.420267 2237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:34.420531 2237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:34.423417 2237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "e5a86043a6d347fc849ea137b7ffb9d0"
format_stamp: "Formatted at 2026-09-24 00:49:34 on dist-test-slave-rtvz"
server_key: "033a083166183ea53df876b370e5e7ad"
server_key_iv: "2a9537f182bd198283021002fcba330b"
server_key_version: "encryptionkey@0"
I20260924 00:49:34.424160 2237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "e5a86043a6d347fc849ea137b7ffb9d0"
format_stamp: "Formatted at 2026-09-24 00:49:34 on dist-test-slave-rtvz"
server_key: "033a083166183ea53df876b370e5e7ad"
server_key_iv: "2a9537f182bd198283021002fcba330b"
server_key_version: "encryptionkey@0"
I20260924 00:49:34.429292 2237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 00:49:34.432689 2252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:34.434183 2237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 00:49:34.434333 2237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "e5a86043a6d347fc849ea137b7ffb9d0"
format_stamp: "Formatted at 2026-09-24 00:49:34 on dist-test-slave-rtvz"
server_key: "033a083166183ea53df876b370e5e7ad"
server_key_iv: "2a9537f182bd198283021002fcba330b"
server_key_version: "encryptionkey@0"
I20260924 00:49:34.434583 2237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:34.459889 2237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:34.464383 2237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:34.464636 2237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:34.467387 2237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:34.467475 2237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:34.467624 2237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:34.467692 2237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:49:34 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:34.492074 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:34.468293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45395 (local address 127.0.222.62:34565)
0924 00:49:34.468462 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:34.468468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:34.470159 (+ 1691us) server_negotiation.cc:407] Connection header received
0924 00:49:34.472503 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:34.472511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:34.472617 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:34.472735 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:34.475051 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.476215 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.478522 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.479218 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.479359 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:34.484407 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:34.484413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:34.484416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:34.484455 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:34.488065 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:34.489332 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:34.489337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:34.489341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:34.489484 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:34.489976 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:34.489983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:34.489987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:34.490401 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:34.490566 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:34.491502 (+ 936us) server_negotiation.cc:299] Negotiation successful
0924 00:49:34.491741 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:49:34.493129 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:34.468993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34565 (local address 127.0.222.2:45395)
0924 00:49:34.469968 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:34.470012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:34.472074 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:34.473059 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:34.473068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:34.473932 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:34.474809 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:34.474833 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.476387 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.476393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:34.478304 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:34.478320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.479529 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.479536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:34.479732 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:34.480597 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:34.480636 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:34.484106 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:34.488274 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:34.488285 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:34.488306 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:34.489137 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:34.489602 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:34.489608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:34.489612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:34.489831 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:34.490560 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:34.490569 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:34.491047 (+ 478us) client_negotiation.cc:769] Sending connection context
0924 00:49:34.491382 (+ 335us) client_negotiation.cc:240] Negotiation successful
0924 00:49:34.491941 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":435,"threads_started":1}
I20260924 00:49:34.507656 2237 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:38119
I20260924 00:49:34.507695 2372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:38119 every 8 connection(s)
I20260924 00:49:34.509153 2237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:49:34.511603 2373 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34565
I20260924 00:49:34.511890 2373 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:34.512825 2373 heartbeater.cc:511] Master 127.0.222.62:34565 requested a full tablet report, sending...
I20260924 00:49:34.514951 2022 ts_manager.cc:194] Registered new tserver with Master: e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119)
I20260924 00:49:34.516101 2022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:45395
I20260924 00:49:34.516255 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 2237
I20260924 00:49:34.516451 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:49:34.516932 888 external_mini_cluster.cc:1428] Setting key 2910221b4c32148f17d25c995acfcd87
I20260924 00:49:34.526549 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:34.500144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:60589 (local address 127.0.222.62:34565)
0924 00:49:34.500343 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:34.500350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:34.515353 (+ 15003us) server_negotiation.cc:407] Connection header received
0924 00:49:34.515529 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:34.515535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:34.515623 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:34.515725 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:34.516834 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.517639 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.519277 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.519683 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.519790 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:34.521602 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:34.521607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:34.521610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:34.521641 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:34.524096 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:34.524703 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:34.524708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:34.524712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:34.524830 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:34.525156 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:34.525161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:34.525164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:34.525508 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:34.525641 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:34.526010 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 00:49:34.526224 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 00:49:34.526640 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:34.500144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34565 (local address 127.0.222.2:60589)
0924 00:49:34.515215 (+ 15071us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:34.515236 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:34.515376 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:34.515749 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:34.515754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:34.516296 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:34.516658 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:34.516671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.517813 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.517817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:34.519150 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:34.519162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.519928 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.519935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:34.520067 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:34.520618 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:34.520635 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:34.521417 (+ 782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:34.524215 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:34.524221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:34.524225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:34.524597 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:34.524931 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:34.524936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:34.524939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:34.525059 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:34.525624 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:34.525628 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:34.525791 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 00:49:34.525908 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 00:49:34.526210 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14904,"thread_start_us":15316,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:49:34.556813 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34565
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:34.878027 2377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:34.878312 2377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:34.878443 2377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:34.885847 2377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:34.885962 2377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:34.886071 2377 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:49:34.886174 2377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:49:34.896330 2377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:34565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:34.897732 2377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:34.899106 2377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:34.912506 2383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:34.912540 2385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:34.912444 2382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:34.913669 2377 server_base.cc:1034] running on GCE node
I20260924 00:49:34.914372 2377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:34.915498 2377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:34.916776 2377 hybrid_clock.cc:648] HybridClock initialized: now 1790210974916699 us; error 50 us; skew 500 ppm
I20260924 00:49:34.917282 2377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:34 dist-test-slave-rtvz krb5kdc[1978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:34.924458 2377 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:49:34.926131 2377 webserver.cc:460] Webserver started at http://127.0.222.3:41479/ using document root <none> and password file <none>
I20260924 00:49:34.926826 2377 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:34.926930 2377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:34.927208 2377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:34.930199 2377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "a4b982faed22419996e9c516f193c0b5"
format_stamp: "Formatted at 2026-09-24 00:49:34 on dist-test-slave-rtvz"
server_key: "1fe982de6370a847a5c595ebaa62584f"
server_key_iv: "8cc756fde5470afe6a037d3873bdf478"
server_key_version: "encryptionkey@0"
I20260924 00:49:34.930938 2377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "a4b982faed22419996e9c516f193c0b5"
format_stamp: "Formatted at 2026-09-24 00:49:34 on dist-test-slave-rtvz"
server_key: "1fe982de6370a847a5c595ebaa62584f"
server_key_iv: "8cc756fde5470afe6a037d3873bdf478"
server_key_version: "encryptionkey@0"
I20260924 00:49:34.936368 2377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:49:34.940089 2392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:34.941543 2377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 00:49:34.941699 2377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "a4b982faed22419996e9c516f193c0b5"
format_stamp: "Formatted at 2026-09-24 00:49:34 on dist-test-slave-rtvz"
server_key: "1fe982de6370a847a5c595ebaa62584f"
server_key_iv: "8cc756fde5470afe6a037d3873bdf478"
server_key_version: "encryptionkey@0"
I20260924 00:49:34.941974 2377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:34.968565 2377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:34.973574 2377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:34.973855 2377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:34.976622 2377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:34.976716 2377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:34.976866 2377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:34.976923 2377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:49:34 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:35.002177 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:34.977433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:53489 (local address 127.0.222.62:34565)
0924 00:49:34.977638 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:34.977643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:34.979120 (+ 1477us) server_negotiation.cc:407] Connection header received
0924 00:49:34.981387 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:34.981393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:34.981525 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:34.981634 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:34.984558 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.985566 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.988531 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.988972 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.989079 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:34.994570 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:34.994577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:34.994583 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:34.994633 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:34.997950 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:34.999066 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:34.999072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:34.999076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:34.999252 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:34.999896 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:34.999902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:34.999905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:35.000295 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:35.000451 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.001552 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.001814 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:49:35.003643 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:34.978014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34565 (local address 127.0.222.3:53489)
0924 00:49:34.978913 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:34.978973 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:34.980920 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:34.982080 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:34.982094 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:34.983073 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:34.984278 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:34.984317 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.985756 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.985761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:34.988336 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:34.988360 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:34.989256 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:34.989265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:34.989628 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:34.990521 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:34.990558 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:34.994233 (+ 3675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:34.998126 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:34.998136 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:34.998160 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:34.998925 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:34.999420 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:34.999426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:34.999430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:34.999765 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:35.000434 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:35.000444 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:35.000988 (+ 544us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.001378 (+ 390us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.002028 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":382,"threads_started":1}
I20260924 00:49:35.012169 2233 heartbeater.cc:503] Master 127.0.222.62:34565 was elected leader, sending a full tablet report...
I20260924 00:49:35.023309 2377 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:41449
I20260924 00:49:35.023365 2513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:41449 every 8 connection(s)
I20260924 00:49:35.024681 2377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:49:35.026966 2514 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34565
I20260924 00:49:35.027382 2514 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:35.027788 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.012437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:56843 (local address 127.0.222.62:34565)
0924 00:49:35.012913 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.012926 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.013247 (+ 321us) server_negotiation.cc:407] Connection header received
0924 00:49:35.013448 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.013453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.013567 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.013697 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:35.015388 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.016342 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.018296 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.018958 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.019083 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.021764 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:35.021771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:35.021776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:35.021823 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:35.024740 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:35.025686 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:35.025691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:35.025694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:35.025864 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:35.026308 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:35.026313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:35.026317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:35.026664 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:35.026802 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.027309 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.027539 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260924 00:49:35.028069 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.012391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34565 (local address 127.0.222.3:56843)
0924 00:49:35.013086 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.013107 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.013307 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.013880 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.013885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.014713 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:35.015170 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.015187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.016491 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.016499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.018120 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.018137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.019393 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.019401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.019542 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.020221 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:35.020239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:35.021542 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:35.024890 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:35.024896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:35.024901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:35.025507 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:35.026019 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:35.026023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:35.026027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:35.026178 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:35.026771 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:35.026776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:35.027006 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.027165 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.027537 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":360,"threads_started":1}
I20260924 00:49:35.028316 2514 heartbeater.cc:511] Master 127.0.222.62:34565 requested a full tablet report, sending...
I20260924 00:49:35.029942 2022 ts_manager.cc:194] Registered new tserver with Master: a4b982faed22419996e9c516f193c0b5 (127.0.222.3:41449)
I20260924 00:49:35.031105 2022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:53489
I20260924 00:49:35.034370 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 2377
I20260924 00:49:35.034557 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:49:35.035094 888 external_mini_cluster.cc:1428] Setting key 35c3a8f4495a826d8fefbfc180487265
I20260924 00:49:35.038753 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:49:35 dist-test-slave-rtvz krb5kdc[1978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210975, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:49:35 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210975, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:35.084066 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.070597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.0.222.62:34565)
0924 00:49:35.070838 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.070844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.070987 (+ 143us) server_negotiation.cc:407] Connection header received
0924 00:49:35.071137 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.071141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.071208 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.071292 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:35.072640 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.073459 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.074684 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.075096 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.075193 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.079336 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:35.079341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:35.079344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:35.079381 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:35.081890 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:35.082466 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:35.082470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:35.082472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:35.082584 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:35.082883 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:35.082886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:35.082888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:35.083137 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:35.083312 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.083558 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.083730 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 00:49:35.088311 2022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:49:35.095048 2022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:49:35.119540 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.108942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:38119 (local address 127.0.0.1:45924)
0924 00:49:35.109556 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.109573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.109683 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.110896 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.110904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.110938 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:35.111382 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.111393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.113408 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.113414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.115984 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.116012 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.118369 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.118376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.118651 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.118711 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.118860 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.119097 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":362,"threads_started":1}
I20260924 00:49:35.120524 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.109373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.0.222.2:38119)
0924 00:49:35.109917 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.109926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.109952 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:49:35.110085 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.110094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.110362 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.110565 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:35.111611 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.113220 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.116214 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.117797 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.117931 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.118193 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.119987 (+ 1794us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.120209 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":255,"threads_started":1}
I20260924 00:49:35.125430 2303 tablet_service.cc:1467] Processing CreateTablet for tablet 35402d02546e46b39a2970b038b396cc (DEFAULT_TABLE table=test-table [id=5b064374ea4044639c6ae7892470a861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:35.127132 2303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35402d02546e46b39a2970b038b396cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.129439 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.108023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.0.222.3:41449)
0924 00:49:35.121034 (+ 13011us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.121043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.121079 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:49:35.121182 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.121191 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.121540 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.122619 (+ 1079us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:35.122932 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.124542 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.127008 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.128429 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.128543 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.128770 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.128953 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.129106 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12811,"thread_start_us":13659,"threads_started":1}
I20260924 00:49:35.129485 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.107865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41449 (local address 127.0.0.1:60082)
0924 00:49:35.108603 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.108639 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.108810 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.121956 (+ 13146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.121961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.121992 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:35.122400 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.122415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.124712 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.124718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.126819 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.126835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.128613 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.128620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.128774 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.128807 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.128895 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.129036 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":34,"thread_start_us":307,"threads_started":1}
I20260924 00:49:35.130659 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.109872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33871 (local address 127.0.0.1:50464)
0924 00:49:35.122213 (+ 12341us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.122238 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.122389 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.123558 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.123563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.123589 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:35.124318 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.124331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.126085 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.126090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.128059 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.128074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.129880 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.129886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.130042 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.130074 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.130170 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.130301 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12185,"mutex_wait_us":1,"thread_start_us":12626,"threads_started":1}
I20260924 00:49:35.130705 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.110162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.0.222.1:33871)
0924 00:49:35.123075 (+ 12913us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.123084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.123109 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:49:35.123184 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.123190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.123374 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.123546 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:35.124537 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.125947 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.128250 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.129691 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.129821 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.130017 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.130220 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.130390 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12772,"thread_start_us":12648,"threads_started":1}
I20260924 00:49:35.135386 2162 tablet_service.cc:1467] Processing CreateTablet for tablet 35402d02546e46b39a2970b038b396cc (DEFAULT_TABLE table=test-table [id=5b064374ea4044639c6ae7892470a861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:35.135386 2443 tablet_service.cc:1467] Processing CreateTablet for tablet 35402d02546e46b39a2970b038b396cc (DEFAULT_TABLE table=test-table [id=5b064374ea4044639c6ae7892470a861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:35.137002 2162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35402d02546e46b39a2970b038b396cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.137022 2443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35402d02546e46b39a2970b038b396cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.140463 2533 tablet_bootstrap.cc:492] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Bootstrap starting.
I20260924 00:49:35.143635 2533 tablet_bootstrap.cc:654] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.144711 2533 log.cc:824] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:35.148396 2533 tablet_bootstrap.cc:492] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: No bootstrap required, opened a new log
I20260924 00:49:35.148675 2533 ts_tablet_manager.cc:1397] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260924 00:49:35.152560 2535 tablet_bootstrap.cc:492] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Bootstrap starting.
I20260924 00:49:35.154562 2536 tablet_bootstrap.cc:492] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Bootstrap starting.
I20260924 00:49:35.156368 2535 tablet_bootstrap.cc:654] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.157827 2536 tablet_bootstrap.cc:654] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.157575 2533 raft_consensus.cc:348] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.158078 2535 log.cc:824] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:35.158651 2533 raft_consensus.cc:374] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.158838 2533 raft_consensus.cc:729] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a86043a6d347fc849ea137b7ffb9d0, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.159197 2536 log.cc:824] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:35.159585 2533 consensus_queue.cc:260] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.161530 2373 heartbeater.cc:503] Master 127.0.222.62:34565 was elected leader, sending a full tablet report...
I20260924 00:49:35.162959 2536 tablet_bootstrap.cc:492] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: No bootstrap required, opened a new log
I20260924 00:49:35.163331 2536 ts_tablet_manager.cc:1397] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 00:49:35.163347 2533 ts_tablet_manager.cc:1428] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260924 00:49:35.169677 2535 tablet_bootstrap.cc:492] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: No bootstrap required, opened a new log
I20260924 00:49:35.169952 2535 ts_tablet_manager.cc:1397] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
I20260924 00:49:35.172977 2536 raft_consensus.cc:348] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.173574 2536 raft_consensus.cc:374] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.173698 2536 raft_consensus.cc:729] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b982faed22419996e9c516f193c0b5, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.174317 2536 consensus_queue.cc:260] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.175885 2514 heartbeater.cc:503] Master 127.0.222.62:34565 was elected leader, sending a full tablet report...
I20260924 00:49:35.175918 2536 ts_tablet_manager.cc:1428] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 00:49:35.178699 2535 raft_consensus.cc:348] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.179037 2535 raft_consensus.cc:374] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.179124 2535 raft_consensus.cc:729] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47910f31b899440ca67a3224006b22b4, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.179703 2535 consensus_queue.cc:260] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.181177 2535 ts_tablet_manager.cc:1428] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260924 00:49:35.218139 2541 raft_consensus.cc:482] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:35.218457 2541 raft_consensus.cc:504] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.220177 2541 leader_election.cc:288] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4b982faed22419996e9c516f193c0b5 (127.0.222.3:41449), e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119)
I20260924 00:49:35.227810 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.220686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:40253 (local address 127.0.222.3:41449)
0924 00:49:35.220866 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.220871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.221543 (+ 672us) server_negotiation.cc:407] Connection header received
0924 00:49:35.221636 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.221642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.221719 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.221825 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:35.222623 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.223707 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.225848 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.226904 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.227000 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.227084 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.227503 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.227609 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:49:35.227982 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.220584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41449 (local address 127.0.222.1:40253)
0924 00:49:35.221329 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.221351 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.221492 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.222009 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.222013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.222050 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:35.222473 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.222487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.223819 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.223825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.225689 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.225702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.227114 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.227121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.227284 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.227320 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.227433 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.227574 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":1,"spinlock_wait_cycles":18688,"thread_start_us":425,"threads_started":1}
I20260924 00:49:35.228940 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.221516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:38119 (local address 127.0.222.1:36501)
0924 00:49:35.222080 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.222092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.222202 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.222521 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.222524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.222542 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:35.223457 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.223469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.224803 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.224807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.226510 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.226524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.228237 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.228243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.228415 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.228448 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.228543 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.228675 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":4,"thread_start_us":280,"threads_started":1}
I20260924 00:49:35.228948 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.221746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:36501 (local address 127.0.222.2:38119)
0924 00:49:35.222033 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.222040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.222163 (+ 123us) server_negotiation.cc:407] Connection header received
0924 00:49:35.222316 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.222321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.222403 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.222497 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:35.223629 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.224655 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.226669 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.227805 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.227951 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.228051 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.228585 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.228716 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 00:49:35.229328 2463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35402d02546e46b39a2970b038b396cc" candidate_uuid: "47910f31b899440ca67a3224006b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b982faed22419996e9c516f193c0b5" is_pre_election: true
I20260924 00:49:35.229933 2463 raft_consensus.cc:2454] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47910f31b899440ca67a3224006b22b4 in term 0.
I20260924 00:49:35.230037 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35402d02546e46b39a2970b038b396cc" candidate_uuid: "47910f31b899440ca67a3224006b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" is_pre_election: true
I20260924 00:49:35.230530 2323 raft_consensus.cc:2454] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47910f31b899440ca67a3224006b22b4 in term 0.
I20260924 00:49:35.230623 2112 leader_election.cc:302] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47910f31b899440ca67a3224006b22b4, a4b982faed22419996e9c516f193c0b5; no voters:
I20260924 00:49:35.231037 2541 raft_consensus.cc:2781] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:35.231195 2541 raft_consensus.cc:482] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:35.231313 2541 raft_consensus.cc:3037] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.233875 2541 raft_consensus.cc:504] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.234375 2541 leader_election.cc:288] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [CANDIDATE]: Term 1 election: Requested vote from peers a4b982faed22419996e9c516f193c0b5 (127.0.222.3:41449), e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119)
I20260924 00:49:35.234905 2463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35402d02546e46b39a2970b038b396cc" candidate_uuid: "47910f31b899440ca67a3224006b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b982faed22419996e9c516f193c0b5"
I20260924 00:49:35.234941 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35402d02546e46b39a2970b038b396cc" candidate_uuid: "47910f31b899440ca67a3224006b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5a86043a6d347fc849ea137b7ffb9d0"
I20260924 00:49:35.235132 2463 raft_consensus.cc:3037] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.235132 2323 raft_consensus.cc:3037] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.238708 2323 raft_consensus.cc:2454] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47910f31b899440ca67a3224006b22b4 in term 1.
I20260924 00:49:35.238708 2463 raft_consensus.cc:2454] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47910f31b899440ca67a3224006b22b4 in term 1.
I20260924 00:49:35.239105 2112 leader_election.cc:302] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47910f31b899440ca67a3224006b22b4, e5a86043a6d347fc849ea137b7ffb9d0; no voters:
I20260924 00:49:35.239396 2541 raft_consensus.cc:2781] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:35.239841 2541 raft_consensus.cc:686] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 1 LEADER]: Becoming Leader. State: Replica: 47910f31b899440ca67a3224006b22b4, State: Running, Role: LEADER
I20260924 00:49:35.240368 2541 consensus_queue.cc:237] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } }
I20260924 00:49:35.245081 2022 catalog_manager.cc:5495] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47910f31b899440ca67a3224006b22b4 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } health_report { overall_health: HEALTHY } } }
W20260924 00:49:35.254380 2234 tablet.cc:2369] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:49:35.264607 2374 tablet.cc:2369] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:49:35.276368 2515 tablet.cc:2369] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:49:35.280807 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.275567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.0.222.1:33871)
0924 00:49:35.275772 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.275776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.275844 (+ 68us) server_negotiation.cc:407] Connection header received
0924 00:49:35.276093 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.276097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.276194 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.276304 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:35.276751 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.277487 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.278767 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.279162 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.279248 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.279700 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:35.280222 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:35.280322 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.280480 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.280545 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:49:35.289371 2323 raft_consensus.cc:1261] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Refusing update from remote peer 47910f31b899440ca67a3224006b22b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:35.289371 2463 raft_consensus.cc:1261] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Refusing update from remote peer 47910f31b899440ca67a3224006b22b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:35.290432 2544 consensus_queue.cc:1035] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:35.290843 2541 consensus_queue.cc:1035] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:35.328562 2022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:35.328832 2022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:35.332502 2022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d: Sending DeleteTablet for 3 replicas of tablet 35402d02546e46b39a2970b038b396cc
I20260924 00:49:35.333878 2303 tablet_service.cc:1514] Processing DeleteTablet for tablet 35402d02546e46b39a2970b038b396cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:35 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:45924
I20260924 00:49:35.333907 2162 tablet_service.cc:1514] Processing DeleteTablet for tablet 35402d02546e46b39a2970b038b396cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:35 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:50464
I20260924 00:49:35.334674 2443 tablet_service.cc:1514] Processing DeleteTablet for tablet 35402d02546e46b39a2970b038b396cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:35 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:60082
I20260924 00:49:35.339775 2022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:49:35.341897 2022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:49:35.346608 2558 tablet_replica.cc:330] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: stopping tablet replica
I20260924 00:49:35.347237 2558 raft_consensus.cc:2229] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:35.349407 2558 raft_consensus.cc:2258] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:35.353264 2558 ts_tablet_manager.cc:1905] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:35.355917 2161 tablet_service.cc:1467] Processing CreateTablet for tablet be61d25e39b64aca979064e192f1f18f (DEFAULT_TABLE table=test-table [id=268c639063bc416ba6e737c03f306309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:35.357012 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be61d25e39b64aca979064e192f1f18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.358139 2559 tablet_replica.cc:330] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: stopping tablet replica
I20260924 00:49:35.358672 2559 raft_consensus.cc:2229] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:35.359750 2559 raft_consensus.cc:2258] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:35.360677 2557 tablet_replica.cc:330] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: stopping tablet replica
I20260924 00:49:35.361370 2557 raft_consensus.cc:2229] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:49:35.361449 2443 tablet_service.cc:1467] Processing CreateTablet for tablet be61d25e39b64aca979064e192f1f18f (DEFAULT_TABLE table=test-table [id=268c639063bc416ba6e737c03f306309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:35.361914 2443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be61d25e39b64aca979064e192f1f18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.362377 2558 ts_tablet_manager.cc:1918] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:49:35.362627 2558 log.cc:1196] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/35402d02546e46b39a2970b038b396cc
I20260924 00:49:35.362555 2303 tablet_service.cc:1467] Processing CreateTablet for tablet be61d25e39b64aca979064e192f1f18f (DEFAULT_TABLE table=test-table [id=268c639063bc416ba6e737c03f306309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:35.363162 2303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be61d25e39b64aca979064e192f1f18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.363504 2558 ts_tablet_manager.cc:1939] T 35402d02546e46b39a2970b038b396cc P e5a86043a6d347fc849ea137b7ffb9d0: Deleting consensus metadata
I20260924 00:49:35.366003 2008 catalog_manager.cc:4841] TS e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119): tablet 35402d02546e46b39a2970b038b396cc (table test-table [id=5b064374ea4044639c6ae7892470a861]) successfully deleted
I20260924 00:49:35.366719 2557 raft_consensus.cc:2258] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:35.369485 2559 ts_tablet_manager.cc:1905] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:35.369877 2557 ts_tablet_manager.cc:1905] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:35.371645 2533 tablet_bootstrap.cc:492] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: Bootstrap starting.
I20260924 00:49:35.374389 2533 tablet_bootstrap.cc:654] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.376158 2536 tablet_bootstrap.cc:492] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: Bootstrap starting.
I20260924 00:49:35.378857 2536 tablet_bootstrap.cc:654] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.381485 2557 ts_tablet_manager.cc:1918] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:49:35.381786 2557 log.cc:1196] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/35402d02546e46b39a2970b038b396cc
I20260924 00:49:35.382876 2557 ts_tablet_manager.cc:1939] T 35402d02546e46b39a2970b038b396cc P 47910f31b899440ca67a3224006b22b4: Deleting consensus metadata
I20260924 00:49:35.384984 2535 tablet_bootstrap.cc:492] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: Bootstrap starting.
I20260924 00:49:35.388057 2535 tablet_bootstrap.cc:654] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.389292 2559 ts_tablet_manager.cc:1918] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:49:35.389489 2536 tablet_bootstrap.cc:492] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: No bootstrap required, opened a new log
I20260924 00:49:35.389549 2559 log.cc:1196] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/35402d02546e46b39a2970b038b396cc
I20260924 00:49:35.389904 2536 ts_tablet_manager.cc:1397] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260924 00:49:35.390422 2559 ts_tablet_manager.cc:1939] T 35402d02546e46b39a2970b038b396cc P a4b982faed22419996e9c516f193c0b5: Deleting consensus metadata
I20260924 00:49:35.391340 2536 raft_consensus.cc:348] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.391711 2536 raft_consensus.cc:374] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.391932 2536 raft_consensus.cc:729] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b982faed22419996e9c516f193c0b5, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.392400 2536 consensus_queue.cc:260] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.392649 2008 catalog_manager.cc:4841] TS a4b982faed22419996e9c516f193c0b5 (127.0.222.3:41449): tablet 35402d02546e46b39a2970b038b396cc (table test-table [id=5b064374ea4044639c6ae7892470a861]) successfully deleted
I20260924 00:49:35.395889 2536 ts_tablet_manager.cc:1428] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 00:49:35.396401 2007 catalog_manager.cc:4841] TS 47910f31b899440ca67a3224006b22b4 (127.0.222.1:33871): tablet 35402d02546e46b39a2970b038b396cc (table test-table [id=5b064374ea4044639c6ae7892470a861]) successfully deleted
I20260924 00:49:35.396807 2535 tablet_bootstrap.cc:492] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: No bootstrap required, opened a new log
I20260924 00:49:35.397029 2535 ts_tablet_manager.cc:1397] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260924 00:49:35.398319 2535 raft_consensus.cc:348] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.398594 2535 raft_consensus.cc:374] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.398800 2535 raft_consensus.cc:729] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47910f31b899440ca67a3224006b22b4, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.399214 2535 consensus_queue.cc:260] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.400573 2535 ts_tablet_manager.cc:1428] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:49:35.400811 2533 tablet_bootstrap.cc:492] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: No bootstrap required, opened a new log
I20260924 00:49:35.401016 2533 ts_tablet_manager.cc:1397] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.004s
I20260924 00:49:35.402400 2533 raft_consensus.cc:348] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.402697 2533 raft_consensus.cc:374] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.402899 2533 raft_consensus.cc:729] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a86043a6d347fc849ea137b7ffb9d0, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.403278 2533 consensus_queue.cc:260] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.404093 2533 ts_tablet_manager.cc:1428] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:49:35.410920 2540 raft_consensus.cc:482] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:35.411281 2540 raft_consensus.cc:504] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.414832 2540 leader_election.cc:288] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47910f31b899440ca67a3224006b22b4 (127.0.222.1:33871), e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119)
I20260924 00:49:35.423918 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.414147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33871 (local address 127.0.222.3:58527)
0924 00:49:35.414547 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.414568 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.414696 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.415384 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.415390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.415423 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:35.415781 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.415794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.417836 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.417842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.420547 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.420566 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.422862 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.422869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.423213 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.423252 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.423349 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.423509 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264}
I20260924 00:49:35.424963 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.413786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:58527 (local address 127.0.222.1:33871)
0924 00:49:35.413997 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.414003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.414830 (+ 827us) server_negotiation.cc:407] Connection header received
0924 00:49:35.414916 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.414920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.415008 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.415111 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:35.415974 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.417651 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.420790 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.422453 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.422622 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.422763 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.424385 (+ 1622us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.424521 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:49:35.426235 2182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be61d25e39b64aca979064e192f1f18f" candidate_uuid: "a4b982faed22419996e9c516f193c0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47910f31b899440ca67a3224006b22b4" is_pre_election: true
I20260924 00:49:35.426985 2182 raft_consensus.cc:2454] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b982faed22419996e9c516f193c0b5 in term 0.
I20260924 00:49:35.428017 2393 leader_election.cc:302] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47910f31b899440ca67a3224006b22b4, a4b982faed22419996e9c516f193c0b5; no voters:
I20260924 00:49:35.428589 2540 raft_consensus.cc:2781] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:35.428773 2540 raft_consensus.cc:482] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:35.428922 2540 raft_consensus.cc:3037] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.433406 2540 raft_consensus.cc:504] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.434527 2540 leader_election.cc:288] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [CANDIDATE]: Term 1 election: Requested vote from peers 47910f31b899440ca67a3224006b22b4 (127.0.222.1:33871), e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119)
I20260924 00:49:35.434976 2182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be61d25e39b64aca979064e192f1f18f" candidate_uuid: "a4b982faed22419996e9c516f193c0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47910f31b899440ca67a3224006b22b4"
I20260924 00:49:35.435191 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.414483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35479 (local address 127.0.222.2:38119)
0924 00:49:35.414667 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.414672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.427034 (+ 12362us) server_negotiation.cc:407] Connection header received
0924 00:49:35.427205 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.427211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.427293 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.427393 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:35.428165 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.429622 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.432520 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.433898 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.434028 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.434126 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.434754 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.434860 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 00:49:35.435273 2182 raft_consensus.cc:3037] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.435957 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.415546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:38119 (local address 127.0.222.3:35479)
0924 00:49:35.426887 (+ 11341us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.426907 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.427054 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.427567 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.427573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.427602 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:35.427966 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.427984 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.429815 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.429821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.432333 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.432347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.434087 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.434095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.434494 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.434539 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.435523 (+ 984us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.435629 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10194,"thread_start_us":10930,"threads_started":1}
I20260924 00:49:35.438217 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be61d25e39b64aca979064e192f1f18f" candidate_uuid: "a4b982faed22419996e9c516f193c0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" is_pre_election: true
I20260924 00:49:35.438589 2323 raft_consensus.cc:2454] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b982faed22419996e9c516f193c0b5 in term 0.
I20260924 00:49:35.439543 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be61d25e39b64aca979064e192f1f18f" candidate_uuid: "a4b982faed22419996e9c516f193c0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5a86043a6d347fc849ea137b7ffb9d0"
I20260924 00:49:35.439683 2182 raft_consensus.cc:2454] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b982faed22419996e9c516f193c0b5 in term 1.
I20260924 00:49:35.439762 2323 raft_consensus.cc:3037] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.440233 2393 leader_election.cc:302] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47910f31b899440ca67a3224006b22b4, a4b982faed22419996e9c516f193c0b5; no voters:
I20260924 00:49:35.440527 2540 raft_consensus.cc:2781] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:35.441191 2540 raft_consensus.cc:686] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 1 LEADER]: Becoming Leader. State: Replica: a4b982faed22419996e9c516f193c0b5, State: Running, Role: LEADER
I20260924 00:49:35.441762 2540 consensus_queue.cc:237] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.443262 2323 raft_consensus.cc:2454] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b982faed22419996e9c516f193c0b5 in term 1.
I20260924 00:49:35.446916 2022 catalog_manager.cc:5495] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4b982faed22419996e9c516f193c0b5 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "a4b982faed22419996e9c516f193c0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } health_report { overall_health: HEALTHY } } }
Sep 24 00:49:35 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:35.527591 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.504297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.0.222.62:34565)
0924 00:49:35.504518 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.504524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.505755 (+ 1231us) server_negotiation.cc:407] Connection header received
0924 00:49:35.505918 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.505924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.506003 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.506135 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:35.508510 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.509955 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.512178 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.513072 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.513269 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.518602 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:35.518609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:35.518614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:35.518651 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:35.523211 (+ 4560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:35.524429 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:35.524435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:35.524439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:35.524703 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:35.525260 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:35.525266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:35.525269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:35.525663 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:35.525850 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.526502 (+ 652us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.526831 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":76}
I20260924 00:49:35.527690 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.504459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34565 (local address 127.0.0.1:51504)
0924 00:49:35.505645 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.505661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.505805 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.506158 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.506161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.507551 (+ 1390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:35.508316 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.508334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.510124 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.510129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.512022 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.512044 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.513332 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.513339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.513598 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.514504 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:35.514526 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:35.518365 (+ 3839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:35.523440 (+ 5075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:35.523458 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:35.523469 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:35.524292 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:35.524828 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:35.524833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:35.524837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:35.525079 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:35.525811 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:35.525821 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:35.526085 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.526340 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.526672 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":785,"threads_started":1}
I20260924 00:49:35.534353 2022 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:51504:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:49:35.535414 2022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:49:35.543268 2161 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2d095b16fd40e6997c6db29186928b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=968756b4b6ca46bdab80296a6e8991fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:49:35.543241 2443 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2d095b16fd40e6997c6db29186928b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=968756b4b6ca46bdab80296a6e8991fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:49:35.543547 2303 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2d095b16fd40e6997c6db29186928b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=968756b4b6ca46bdab80296a6e8991fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:49:35.543881 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2d095b16fd40e6997c6db29186928b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.543915 2443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2d095b16fd40e6997c6db29186928b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.544214 2303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2d095b16fd40e6997c6db29186928b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:35.552860 2536 tablet_bootstrap.cc:492] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: Bootstrap starting.
I20260924 00:49:35.553992 2535 tablet_bootstrap.cc:492] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: Bootstrap starting.
I20260924 00:49:35.555615 2536 tablet_bootstrap.cc:654] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.556066 2533 tablet_bootstrap.cc:492] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: Bootstrap starting.
I20260924 00:49:35.557938 2535 tablet_bootstrap.cc:654] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.558820 2533 tablet_bootstrap.cc:654] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:35.567572 2535 tablet_bootstrap.cc:492] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: No bootstrap required, opened a new log
I20260924 00:49:35.567847 2535 ts_tablet_manager.cc:1397] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20260924 00:49:35.568295 2533 tablet_bootstrap.cc:492] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: No bootstrap required, opened a new log
I20260924 00:49:35.568531 2533 ts_tablet_manager.cc:1397] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260924 00:49:35.568972 2535 raft_consensus.cc:348] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.569327 2535 raft_consensus.cc:374] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.569475 2535 raft_consensus.cc:729] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47910f31b899440ca67a3224006b22b4, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.569432 2533 raft_consensus.cc:348] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.569732 2533 raft_consensus.cc:374] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.569854 2533 raft_consensus.cc:729] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5a86043a6d347fc849ea137b7ffb9d0, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.569847 2535 consensus_queue.cc:260] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.570197 2533 consensus_queue.cc:260] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.570426 2552 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.571024 2552 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:35.570791 2539 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.571350 2539 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:35.572583 2533 ts_tablet_manager.cc:1428] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 00:49:35.573922 2536 tablet_bootstrap.cc:492] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: No bootstrap required, opened a new log
I20260924 00:49:35.574186 2536 ts_tablet_manager.cc:1397] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.004s
I20260924 00:49:35.575212 2536 raft_consensus.cc:348] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.575404 2535 ts_tablet_manager.cc:1428] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 00:49:35.575567 2536 raft_consensus.cc:374] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:35.575708 2536 raft_consensus.cc:729] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b982faed22419996e9c516f193c0b5, State: Initialized, Role: FOLLOWER
I20260924 00:49:35.576076 2536 consensus_queue.cc:260] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.576794 2536 ts_tablet_manager.cc:1428] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:49:35.576634 2540 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.577070 2540 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:35.778972 2552 raft_consensus.cc:482] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:35.779193 2552 raft_consensus.cc:504] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.779932 2552 leader_election.cc:288] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119), a4b982faed22419996e9c516f193c0b5 (127.0.222.3:41449)
I20260924 00:49:35.780582 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2d095b16fd40e6997c6db29186928b" candidate_uuid: "47910f31b899440ca67a3224006b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" is_pre_election: true
I20260924 00:49:35.780601 2463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2d095b16fd40e6997c6db29186928b" candidate_uuid: "47910f31b899440ca67a3224006b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b982faed22419996e9c516f193c0b5" is_pre_election: true
I20260924 00:49:35.780840 2463 raft_consensus.cc:2454] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47910f31b899440ca67a3224006b22b4 in term 0.
I20260924 00:49:35.780862 2323 raft_consensus.cc:2454] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47910f31b899440ca67a3224006b22b4 in term 0.
I20260924 00:49:35.781345 2112 leader_election.cc:302] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47910f31b899440ca67a3224006b22b4, a4b982faed22419996e9c516f193c0b5; no voters:
I20260924 00:49:35.781620 2552 raft_consensus.cc:2781] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:35.781770 2552 raft_consensus.cc:482] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:35.781885 2552 raft_consensus.cc:3037] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.785558 2552 raft_consensus.cc:504] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.786410 2552 leader_election.cc:288] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [CANDIDATE]: Term 1 election: Requested vote from peers e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119), a4b982faed22419996e9c516f193c0b5 (127.0.222.3:41449)
I20260924 00:49:35.787004 2323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2d095b16fd40e6997c6db29186928b" candidate_uuid: "47910f31b899440ca67a3224006b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5a86043a6d347fc849ea137b7ffb9d0"
I20260924 00:49:35.787039 2463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2d095b16fd40e6997c6db29186928b" candidate_uuid: "47910f31b899440ca67a3224006b22b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b982faed22419996e9c516f193c0b5"
I20260924 00:49:35.787240 2463 raft_consensus.cc:3037] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.787261 2323 raft_consensus.cc:3037] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:35.790273 2323 raft_consensus.cc:2454] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47910f31b899440ca67a3224006b22b4 in term 1.
I20260924 00:49:35.790402 2463 raft_consensus.cc:2454] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47910f31b899440ca67a3224006b22b4 in term 1.
I20260924 00:49:35.790776 2112 leader_election.cc:302] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47910f31b899440ca67a3224006b22b4, e5a86043a6d347fc849ea137b7ffb9d0; no voters:
I20260924 00:49:35.791155 2552 raft_consensus.cc:2781] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:35.791397 2552 raft_consensus.cc:686] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [term 1 LEADER]: Becoming Leader. State: Replica: 47910f31b899440ca67a3224006b22b4, State: Running, Role: LEADER
I20260924 00:49:35.791802 2552 consensus_queue.cc:237] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } }
I20260924 00:49:35.793982 2541 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: TxnStatusTablet state changed. Reason: New leader 47910f31b899440ca67a3224006b22b4. Latest consensus state: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.794291 2541 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:49:35.795377 2577 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:49:35.796075 2577 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:49:35.795742 2022 catalog_manager.cc:5495] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47910f31b899440ca67a3224006b22b4 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } health_report { overall_health: UNKNOWN } } }
I20260924 00:49:35.836184 2540 consensus_queue.cc:1035] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:35.842470 2540 consensus_queue.cc:1035] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:35.866458 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.859354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.222.1:33871)
0924 00:49:35.859584 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.859612 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.859633 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:49:35.859700 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.859706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.859835 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.859931 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:35.860581 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.862014 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.864127 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.864935 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.865067 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.865750 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:35.865848 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:35.865921 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.866074 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.866139 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 00:49:35.866945 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.858348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33871 (local address 127.0.0.1:50478)
0924 00:49:35.858887 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.858908 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.859053 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.859991 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.859996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.860009 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:49:35.860384 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.860399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.862183 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.862188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.863994 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.864006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.865097 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.865103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.865431 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.865549 (+ 118us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:49:35.865947 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:49:35.865970 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.866066 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.866246 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318}
I20260924 00:49:35.871219 2463 raft_consensus.cc:1261] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Refusing update from remote peer 47910f31b899440ca67a3224006b22b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:35.871454 2323 raft_consensus.cc:1261] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Refusing update from remote peer 47910f31b899440ca67a3224006b22b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:35.871863 2541 consensus_queue.cc:1035] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:35.872169 2552 consensus_queue.cc:1035] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:35.881675 2539 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: TxnStatusTablet state changed. Reason: New leader 47910f31b899440ca67a3224006b22b4. Latest consensus state: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.882145 2539 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:35.883782 2581 mvcc.cc:205] Tried to move back new op lower bound from 7332704157161545728 to 7332704156846456832. Current Snapshot: MvccSnapshot[applied={T|T < 7332704157161545728}]
I20260924 00:49:35.882849 2580 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: TxnStatusTablet state changed. Reason: New leader 47910f31b899440ca67a3224006b22b4. Latest consensus state: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.884503 2580 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:35.889042 2583 mvcc.cc:205] Tried to move back new op lower bound from 7332704157161545728 to 7332704156846456832. Current Snapshot: MvccSnapshot[applied={T|T < 7332704157161545728}]
I20260924 00:49:35.891952 2552 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.892267 2552 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:49:35.892551 2552 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.892903 2552 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P 47910f31b899440ca67a3224006b22b4: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:49:35.897820 2539 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.898093 2539 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P e5a86043a6d347fc849ea137b7ffb9d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:35.901750 2580 tablet_replica.cc:439] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47910f31b899440ca67a3224006b22b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47910f31b899440ca67a3224006b22b4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33871 } } peers { permanent_uuid: "e5a86043a6d347fc849ea137b7ffb9d0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38119 } } peers { permanent_uuid: "a4b982faed22419996e9c516f193c0b5" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41449 } } }
I20260924 00:49:35.902132 2580 tablet_replica.cc:442] T 9f2d095b16fd40e6997c6db29186928b P a4b982faed22419996e9c516f193c0b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:35.926291 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.908741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.0.222.3:41449)
0924 00:49:35.917325 (+ 8584us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.917330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.917359 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:49:35.917461 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.917465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.917548 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.917736 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:35.918990 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.920009 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.923047 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.923672 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.923791 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.924596 (+ 805us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:35.925378 (+ 782us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:35.925492 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.925656 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.925759 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8434,"thread_start_us":8300,"threads_started":1}
I20260924 00:49:35.947002 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.938447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:59213 (local address 127.0.222.1:33871)
0924 00:49:35.938666 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.938674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.939595 (+ 921us) server_negotiation.cc:407] Connection header received
0924 00:49:35.939800 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.939803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.939885 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.939964 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:35.940785 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.942032 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.944611 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.945414 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.945516 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.946159 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:35.946296 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:35.946386 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.946605 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.946679 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 00:49:35.947412 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.938432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33871 (local address 127.0.222.3:59213)
0924 00:49:35.939488 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.939510 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.939677 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.940018 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.940023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.940037 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:49:35.940548 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.940563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.942198 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.942204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.944441 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.944460 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.945602 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.945609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.945881 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.945990 (+ 109us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:49:35.946446 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:49:35.946461 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.946591 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.946762 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":533,"threads_started":1}
I20260924 00:49:35.997136 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.987776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:53977 (local address 127.0.222.3:41449)
0924 00:49:35.987932 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:35.987939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:35.989740 (+ 1801us) server_negotiation.cc:407] Connection header received
0924 00:49:35.989969 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:35.989974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:35.990077 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:35.990154 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:35.991172 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.992143 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.994668 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.995407 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.995504 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.996223 (+ 719us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:35.996329 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:35.996404 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:35.996584 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 00:49:35.996665 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:49:35.997637 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:35.988077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41449 (local address 127.0.222.1:53977)
0924 00:49:35.989616 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:35.989638 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:35.989815 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:35.990188 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:35.990192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:35.990204 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:49:35.990964 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.990980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.992293 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.992298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.994499 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:35.994515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:35.995724 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:35.995729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:35.995960 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:35.996023 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:49:35.996432 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:49:35.996444 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 00:49:35.996555 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 00:49:35.996701 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"thread_start_us":1223,"threads_started":1}
I20260924 00:49:36.066038 2022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:36.066354 2022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:36.070616 2022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7b2bbbb4f174595b4eda491a1f0942d: Sending DeleteTablet for 3 replicas of tablet be61d25e39b64aca979064e192f1f18f
I20260924 00:49:36.071894 2303 tablet_service.cc:1514] Processing DeleteTablet for tablet be61d25e39b64aca979064e192f1f18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:36 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:45924
I20260924 00:49:36.072304 2443 tablet_service.cc:1514] Processing DeleteTablet for tablet be61d25e39b64aca979064e192f1f18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:36 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:60082
I20260924 00:49:36.072350 2161 tablet_service.cc:1514] Processing DeleteTablet for tablet be61d25e39b64aca979064e192f1f18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:36 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:50464
I20260924 00:49:36.086063 2604 tablet_replica.cc:330] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: stopping tablet replica
I20260924 00:49:36.086373 2604 raft_consensus.cc:2229] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:36.086944 2604 raft_consensus.cc:2258] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:36.088425 2604 ts_tablet_manager.cc:1905] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:36.091733 2603 tablet_replica.cc:330] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: stopping tablet replica
I20260924 00:49:36.092142 2603 raft_consensus.cc:2229] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:36.092612 2603 raft_consensus.cc:2258] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:36.093847 2603 ts_tablet_manager.cc:1905] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:36.097795 2604 ts_tablet_manager.cc:1918] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:36.098488 2604 log.cc:1196] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/be61d25e39b64aca979064e192f1f18f
I20260924 00:49:36.099522 2605 tablet_replica.cc:330] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: stopping tablet replica
I20260924 00:49:36.099921 2605 raft_consensus.cc:2229] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:49:36.100816 2605 raft_consensus.cc:2258] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:36.102072 2605 ts_tablet_manager.cc:1905] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:36.102177 2603 ts_tablet_manager.cc:1918] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:36.106711 2604 ts_tablet_manager.cc:1939] T be61d25e39b64aca979064e192f1f18f P 47910f31b899440ca67a3224006b22b4: Deleting consensus metadata
I20260924 00:49:36.108521 2603 log.cc:1196] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/be61d25e39b64aca979064e192f1f18f
I20260924 00:49:36.109540 2007 catalog_manager.cc:4841] TS 47910f31b899440ca67a3224006b22b4 (127.0.222.1:33871): tablet be61d25e39b64aca979064e192f1f18f (table test-table [id=268c639063bc416ba6e737c03f306309]) successfully deleted
I20260924 00:49:36.110122 2603 ts_tablet_manager.cc:1939] T be61d25e39b64aca979064e192f1f18f P e5a86043a6d347fc849ea137b7ffb9d0: Deleting consensus metadata
I20260924 00:49:36.113857 2605 ts_tablet_manager.cc:1918] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:36.114143 2605 log.cc:1196] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/be61d25e39b64aca979064e192f1f18f
I20260924 00:49:36.114786 2605 ts_tablet_manager.cc:1939] T be61d25e39b64aca979064e192f1f18f P a4b982faed22419996e9c516f193c0b5: Deleting consensus metadata
I20260924 00:49:36.115549 2008 catalog_manager.cc:4841] TS e5a86043a6d347fc849ea137b7ffb9d0 (127.0.222.2:38119): tablet be61d25e39b64aca979064e192f1f18f (table test-table [id=268c639063bc416ba6e737c03f306309]) successfully deleted
I20260924 00:49:36.115900 2008 catalog_manager.cc:4841] TS a4b982faed22419996e9c516f193c0b5 (127.0.222.3:41449): tablet be61d25e39b64aca979064e192f1f18f (table test-table [id=268c639063bc416ba6e737c03f306309]) successfully deleted
Sep 24 00:49:36 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210975, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:49:36.132619 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:36.091765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.0.222.1:33871)
0924 00:49:36.091977 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:36.091984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:36.103387 (+ 11403us) server_negotiation.cc:407] Connection header received
0924 00:49:36.103752 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:36.103759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:36.103880 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:36.103963 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:36.110376 (+ 6413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.111883 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.116665 (+ 4782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.117705 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.117861 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:36.123920 (+ 6059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:36.123941 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:36.123962 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:36.124022 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:36.128425 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.129600 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.129612 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.129615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.129914 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.130476 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.130482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.130485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.131042 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:36.131457 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:36.131797 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 00:49:36.132069 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 00:49:36.133695 2228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50488
I20260924 00:49:36.162156 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:36.138465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.0.222.62:34565)
0924 00:49:36.139335 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:36.139342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:36.140272 (+ 930us) server_negotiation.cc:407] Connection header received
0924 00:49:36.140501 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:36.140506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:36.140598 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:36.140720 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:36.143249 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.144697 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.147094 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.147928 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.148073 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:36.152379 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:36.152388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:36.152393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:36.152441 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:36.157287 (+ 4846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.158297 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.158303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.158306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.158492 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.158988 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.158993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.158997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.159502 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:36.159802 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:36.161326 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0924 00:49:36.161589 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":741,"threads_started":1}
W20260924 00:49:36.163220 2022 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51518
I20260924 00:49:36.165897 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2095
I20260924 00:49:36.187894 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:36.167858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.0.222.1:33871)
0924 00:49:36.168045 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:36.168051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:36.169303 (+ 1252us) server_negotiation.cc:407] Connection header received
0924 00:49:36.169507 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:36.169513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:36.169612 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:36.169713 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:36.171223 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.172476 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.174435 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.175194 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.175300 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:36.179204 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:36.179211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:36.179216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:36.179262 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:36.183864 (+ 4602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.184915 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.184921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.184924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.185185 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.185687 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.185692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.185695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.186120 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:36.186500 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:36.186809 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 00:49:36.187083 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 00:49:36.188642 2228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50500
W20260924 00:49:36.189392 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2095: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:36.189669 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 2095
I20260924 00:49:36.208858 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2237
Sep 24 00:49:36 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210975, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:49:36.227694 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:36.209779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.0.222.2:38119)
0924 00:49:36.210617 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:36.210621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:36.210638 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:49:36.210692 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:36.210696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:36.210777 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:36.210887 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:36.212788 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.214020 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.215901 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.216484 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.216586 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:36.221701 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:36.221722 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:36.221744 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:36.221791 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:36.224625 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.225498 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.225506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.225509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.225651 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.226130 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.226134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.226136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.226523 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:36.226796 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:36.227106 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 00:49:36.227329 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":538,"threads_started":1}
W20260924 00:49:36.228441 2368 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45930
W20260924 00:49:36.229279 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:36.229472 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 2237
I20260924 00:49:36.243428 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2377
Sep 24 00:49:36 dist-test-slave-rtvz krb5kdc[1978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210975, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:49:36.260780 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:36.244238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.0.222.3:41449)
0924 00:49:36.244422 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:36.244428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:36.244453 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:49:36.244616 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:36.244621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:36.244720 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:36.244880 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:36.246283 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.247521 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.248849 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:36.249545 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:36.249676 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:36.253731 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:36.253750 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:36.253773 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:36.253817 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:36.257742 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.258493 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.258503 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.258507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.258662 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:36.259045 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:36.259051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:36.259054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:36.259527 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:36.259882 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:36.260222 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 00:49:36.260485 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 00:49:36.261669 2505 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60092
W20260924 00:49:36.262343 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2377: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:36.262486 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 2377
I20260924 00:49:36.276894 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 1991
I20260924 00:49:36.278136 2064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:36.392483 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 1991
2026-09-24T00:49:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4592 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:36 dist-test-slave-rtvz krb5kdc[2616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:36 dist-test-slave-rtvz krb5kdc[2616](info): set up 2 sockets
Sep 24 00:49:36 dist-test-slave-rtvz krb5kdc[2616](info): commencing operation
krb5kdc: starting...
W20260924 00:49:37.413231 888 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.011s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:49:37 dist-test-slave-rtvz krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:49:37.593044 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45591
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:37.923219 2629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:37.923547 2629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:37.923666 2629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:37.931414 2629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:49:37.931559 2629 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:49:37.931634 2629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:37.931711 2629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:37.931780 2629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:49:37.931851 2629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:49:37.942469 2629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45591
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:37.943920 2629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:37.945451 2629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:37.959259 2637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:37.959511 2634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:37.959347 2635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:37.960503 2629 server_base.cc:1034] running on GCE node
I20260924 00:49:37.961585 2629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:37.963734 2629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:37.965130 2629 hybrid_clock.cc:648] HybridClock initialized: now 1790210977965021 us; error 82 us; skew 500 ppm
I20260924 00:49:37.965628 2629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:37 dist-test-slave-rtvz krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:37.969589 2629 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:49:37.971256 2629 webserver.cc:460] Webserver started at http://127.0.222.62:38577/ using document root <none> and password file <none>
I20260924 00:49:37.972021 2629 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:37.972127 2629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:37.972419 2629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:37.976053 2629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "de5a7bd8e5244a57bf77787775ad2143"
format_stamp: "Formatted at 2026-09-24 00:49:37 on dist-test-slave-rtvz"
server_key: "a2035799e0c87be6128afb0f6d8c5215"
server_key_iv: "e6af74211626915bc7334a50924e573d"
server_key_version: "encryptionkey@0"
I20260924 00:49:37.976900 2629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "de5a7bd8e5244a57bf77787775ad2143"
format_stamp: "Formatted at 2026-09-24 00:49:37 on dist-test-slave-rtvz"
server_key: "a2035799e0c87be6128afb0f6d8c5215"
server_key_iv: "e6af74211626915bc7334a50924e573d"
server_key_version: "encryptionkey@0"
I20260924 00:49:37.982699 2629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:49:37.986713 2644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:37.988374 2629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:37.988590 2629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "de5a7bd8e5244a57bf77787775ad2143"
format_stamp: "Formatted at 2026-09-24 00:49:37 on dist-test-slave-rtvz"
server_key: "a2035799e0c87be6128afb0f6d8c5215"
server_key_iv: "e6af74211626915bc7334a50924e573d"
server_key_version: "encryptionkey@0"
I20260924 00:49:37.988797 2629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:38.008538 2629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:38.013348 2629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:38.013607 2629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:38.035712 2629 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45591
I20260924 00:49:38.035735 2706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45591 every 8 connection(s)
I20260924 00:49:38.037120 2629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:49:38.041361 2707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:38.041500 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 2629
I20260924 00:49:38.041702 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:49:38.042318 888 external_mini_cluster.cc:1428] Setting key 88297db3cae251cc38a0d12547a6783f
I20260924 00:49:38.055670 2707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: Bootstrap starting.
Sep 24 00:49:38 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:38.059734 2707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:38.061144 2707 log.cc:824] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:38.064711 2707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: No bootstrap required, opened a new log
I20260924 00:49:38.068964 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:38.044345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.222.62:45591)
0924 00:49:38.045249 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:38.045265 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:38.045310 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:49:38.047609 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:38.047636 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:38.048176 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:38.048545 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:38.050413 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.051910 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.053801 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.054551 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.054697 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:38.059502 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:38.059514 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:38.059537 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:38.059598 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:38.063340 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:38.064254 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:38.064263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:38.064273 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:38.064520 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:38.065031 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:38.065038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:38.065041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:38.065963 (+ 922us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:38.066284 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:38.066753 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 00:49:38.067360 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":439,"threads_started":1}
I20260924 00:49:38.074534 2707 raft_consensus.cc:348] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5a7bd8e5244a57bf77787775ad2143" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45591 } }
I20260924 00:49:38.074946 2707 raft_consensus.cc:374] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:38.075033 2707 raft_consensus.cc:729] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5a7bd8e5244a57bf77787775ad2143, State: Initialized, Role: FOLLOWER
I20260924 00:49:38.075634 2707 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5a7bd8e5244a57bf77787775ad2143" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45591 } }
I20260924 00:49:38.075905 2707 raft_consensus.cc:388] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:49:38.076061 2707 raft_consensus.cc:482] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:49:38.076258 2707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:38.078665 2707 raft_consensus.cc:504] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5a7bd8e5244a57bf77787775ad2143" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45591 } }
I20260924 00:49:38.079241 2707 leader_election.cc:302] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de5a7bd8e5244a57bf77787775ad2143; no voters:
I20260924 00:49:38.079947 2707 leader_election.cc:288] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:49:38.080022 2712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:38.080729 2712 raft_consensus.cc:686] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [term 1 LEADER]: Becoming Leader. State: Replica: de5a7bd8e5244a57bf77787775ad2143, State: Running, Role: LEADER
I20260924 00:49:38.081301 2712 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5a7bd8e5244a57bf77787775ad2143" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45591 } }
I20260924 00:49:38.082163 2707 sys_catalog.cc:564] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:49:38.084282 2714 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5a7bd8e5244a57bf77787775ad2143. Latest consensus state: current_term: 1 leader_uuid: "de5a7bd8e5244a57bf77787775ad2143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5a7bd8e5244a57bf77787775ad2143" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45591 } } }
I20260924 00:49:38.084657 2714 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:38.084455 2713 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de5a7bd8e5244a57bf77787775ad2143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5a7bd8e5244a57bf77787775ad2143" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45591 } } }
I20260924 00:49:38.084849 2713 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143 [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:38.086983 2718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:49:38.092741 2718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:49:38.103034 2718 catalog_manager.cc:1347] Generated new cluster ID: c998ff1ebf374fc99ebb14fdb411b26e
I20260924 00:49:38.103158 2718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:49:38.124461 2718 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:49:38.125962 2718 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:49:38.146112 2718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: Generated new TSK 0
I20260924 00:49:38.147163 2718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:49:38.196439 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45591
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:38.524520 2733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:38.524802 2733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:38.524943 2733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:38.532168 2733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:38.532284 2733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:38.532398 2733 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:49:38.532467 2733 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:49:38.532549 2733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:49:38.542516 2733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:38.543937 2733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:38.545284 2733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:38.558970 2738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:38.559039 2741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:38.559010 2739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:38.559721 2733 server_base.cc:1034] running on GCE node
I20260924 00:49:38.560379 2733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:38.561415 2733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:38.562631 2733 hybrid_clock.cc:648] HybridClock initialized: now 1790210978562563 us; error 60 us; skew 500 ppm
I20260924 00:49:38.562978 2733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:38 dist-test-slave-rtvz krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:38.566445 2733 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:49:38.568147 2733 webserver.cc:460] Webserver started at http://127.0.222.1:42635/ using document root <none> and password file <none>
I20260924 00:49:38.568902 2733 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:38.569036 2733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:38.569396 2733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:38.572410 2733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "3ac3deb94f5d48a1b5106e76eef69daa"
format_stamp: "Formatted at 2026-09-24 00:49:38 on dist-test-slave-rtvz"
server_key: "e9e28edac33027ddbbecf8df8176954d"
server_key_iv: "3311203f458c08fb5f3a4a39df896970"
server_key_version: "encryptionkey@0"
I20260924 00:49:38.573216 2733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "3ac3deb94f5d48a1b5106e76eef69daa"
format_stamp: "Formatted at 2026-09-24 00:49:38 on dist-test-slave-rtvz"
server_key: "e9e28edac33027ddbbecf8df8176954d"
server_key_iv: "3311203f458c08fb5f3a4a39df896970"
server_key_version: "encryptionkey@0"
I20260924 00:49:38.578737 2733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 00:49:38.582537 2748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:38.584091 2733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:49:38.584264 2733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "3ac3deb94f5d48a1b5106e76eef69daa"
format_stamp: "Formatted at 2026-09-24 00:49:38 on dist-test-slave-rtvz"
server_key: "e9e28edac33027ddbbecf8df8176954d"
server_key_iv: "3311203f458c08fb5f3a4a39df896970"
server_key_version: "encryptionkey@0"
I20260924 00:49:38.584468 2733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:38.617892 2733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:38.622166 2733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:38.622437 2733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:38.625063 2733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:38.625166 2733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:38.625298 2733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:38.625371 2733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:49:38 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:38.648658 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:38.625801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45379 (local address 127.0.222.62:45591)
0924 00:49:38.626255 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:38.626262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:38.627458 (+ 1196us) server_negotiation.cc:407] Connection header received
0924 00:49:38.630516 (+ 3058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:38.630523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:38.630617 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:38.630734 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:38.633062 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.634264 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.636464 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.637049 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.637178 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:38.642094 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:38.642100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:38.642103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:38.642139 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:38.645246 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:38.646190 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:38.646197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:38.646200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:38.646359 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:38.646819 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:38.646825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:38.646829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:38.647155 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:38.647290 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:38.648027 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 00:49:38.648262 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":226,"threads_started":1}
I20260924 00:49:38.650138 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:38.626432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45591 (local address 127.0.222.1:45379)
0924 00:49:38.627262 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:38.627314 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:38.630080 (+ 2766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:38.631208 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:38.631221 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:38.632003 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:38.632859 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:38.632886 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.634446 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.634452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:38.636303 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:38.636323 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.637453 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.637462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:38.637678 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:38.638364 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:38.638387 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:38.641808 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:38.645447 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:38.645456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:38.645470 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:38.646039 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:38.646475 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:38.646481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:38.646485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:38.646706 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:38.647283 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:38.647292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:38.647676 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 00:49:38.647922 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 00:49:38.648508 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":336,"threads_started":1}
I20260924 00:49:38.669896 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:38.656148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:43557 (local address 127.0.222.62:45591)
0924 00:49:38.656360 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:38.656366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:38.657161 (+ 795us) server_negotiation.cc:407] Connection header received
0924 00:49:38.657427 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:38.657432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:38.657502 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:38.657605 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:38.658577 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.659358 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.660660 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.661063 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.661169 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:38.663585 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:38.663592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:38.663597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:38.663639 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:38.666986 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:38.667858 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:38.667863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:38.667867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:38.668049 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:38.668472 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:38.668478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:38.668481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:38.668748 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:38.668884 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:38.669335 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 00:49:38.669558 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 00:49:38.670001 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:38.656161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45591 (local address 127.0.222.1:43557)
0924 00:49:38.657032 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:38.657062 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:38.657261 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:38.657670 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:38.657676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:38.658141 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:38.658436 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:38.658447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.659493 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.659498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:38.660532 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:38.660544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:38.661260 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:38.661267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:38.661454 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:38.662187 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:38.662204 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:38.663366 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:38.667138 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:38.667143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:38.667147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:38.667740 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:38.668189 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:38.668194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:38.668198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:38.668356 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:38.668872 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:38.668878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:38.669059 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 00:49:38.669187 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 00:49:38.669525 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":536,"threads_started":1}
I20260924 00:49:38.677441 2733 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:41785
I20260924 00:49:38.677508 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:41785 every 8 connection(s)
I20260924 00:49:38.678711 2733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:49:38.680835 2871 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45591
I20260924 00:49:38.681072 2871 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:38.681818 2871 heartbeater.cc:511] Master 127.0.222.62:45591 requested a full tablet report, sending...
I20260924 00:49:38.683954 2661 ts_manager.cc:194] Registered new tserver with Master: 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785)
I20260924 00:49:38.684499 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 2733
I20260924 00:49:38.684660 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:49:38.685369 888 external_mini_cluster.cc:1428] Setting key c3c8a4f0e91a0df791c6d2f5ab5cbf67
I20260924 00:49:38.685508 2661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:45379
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:49:38.725370 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45591
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:39.052759 2875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:39.053151 2875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:39.053325 2875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:39.060724 2875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:39.060853 2875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:39.060988 2875 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:49:39.061055 2875 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:49:39.061132 2875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:49:39.071287 2875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:39.072656 2875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:39.074072 2875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:39.088126 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:39.088156 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:39.088560 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:39.089053 2875 server_base.cc:1034] running on GCE node
I20260924 00:49:39.089756 2875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:39.090740 2875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:39.091955 2875 hybrid_clock.cc:648] HybridClock initialized: now 1790210979091901 us; error 42 us; skew 500 ppm
I20260924 00:49:39.092283 2875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:39 dist-test-slave-rtvz krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:39.095592 2875 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:49:39.097190 2875 webserver.cc:460] Webserver started at http://127.0.222.2:39077/ using document root <none> and password file <none>
I20260924 00:49:39.097923 2875 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:39.098034 2875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:39.098315 2875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:39.101321 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "534411c11bc44528aad2fe49b731f91c"
format_stamp: "Formatted at 2026-09-24 00:49:39 on dist-test-slave-rtvz"
server_key: "7e7769aff48647c45e91f0de50cc2156"
server_key_iv: "d3d6ee7952051314a79c60e396df2282"
server_key_version: "encryptionkey@0"
I20260924 00:49:39.102149 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "534411c11bc44528aad2fe49b731f91c"
format_stamp: "Formatted at 2026-09-24 00:49:39 on dist-test-slave-rtvz"
server_key: "7e7769aff48647c45e91f0de50cc2156"
server_key_iv: "d3d6ee7952051314a79c60e396df2282"
server_key_version: "encryptionkey@0"
I20260924 00:49:39.107527 2875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 00:49:39.111145 2890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:39.112699 2875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:49:39.112879 2875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "534411c11bc44528aad2fe49b731f91c"
format_stamp: "Formatted at 2026-09-24 00:49:39 on dist-test-slave-rtvz"
server_key: "7e7769aff48647c45e91f0de50cc2156"
server_key_iv: "d3d6ee7952051314a79c60e396df2282"
server_key_version: "encryptionkey@0"
I20260924 00:49:39.113095 2875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:39.128297 2875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:39.132975 2875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:39.133327 2875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:39.136351 2875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:39.136488 2875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:39.136683 2875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:39.136780 2875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:49:39 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:39.161078 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.136696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:43025 (local address 127.0.222.62:45591)
0924 00:49:39.136871 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.136878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.138534 (+ 1656us) server_negotiation.cc:407] Connection header received
0924 00:49:39.140556 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.140563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.140677 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.140814 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:39.142864 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.143857 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.147107 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.147653 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.147797 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.153367 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:39.153376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:39.153382 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:39.153433 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:39.156986 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.158138 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.158144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.158148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.158338 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.158855 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.158861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.158865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.159169 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:39.159334 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.160581 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.160827 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:49:39.163033 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.137441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45591 (local address 127.0.222.2:43025)
0924 00:49:39.138353 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.138390 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.140227 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.140997 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.141005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.141678 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:39.142665 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.142690 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.144035 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.144043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.146909 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.146933 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.147954 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.147964 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.148367 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.149299 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:39.149338 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:39.152905 (+ 3567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:39.157215 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.157228 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.157248 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.157924 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.158484 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.158492 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.158496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.158722 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.159369 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:39.159383 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:39.160027 (+ 644us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.160450 (+ 423us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.161153 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":417,"threads_started":1}
I20260924 00:49:39.181932 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.168875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:47371 (local address 127.0.222.62:45591)
0924 00:49:39.169073 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.169080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.169813 (+ 733us) server_negotiation.cc:407] Connection header received
0924 00:49:39.170068 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.170073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.170144 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.170248 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:39.171200 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.171971 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.173408 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.173965 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.174080 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.175856 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:39.175861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:39.175864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:39.175898 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:39.178868 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.179816 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.179821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.179824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.179993 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.180420 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.180425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.180429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.180761 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:39.180905 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.181359 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.181585 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:49:39.182053 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.168853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45591 (local address 127.0.222.2:47371)
0924 00:49:39.169691 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.169715 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.169902 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.170303 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.170308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.170779 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:39.171057 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.171066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.172111 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.172117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.173260 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.173274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.174163 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.174170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.174333 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.174918 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:39.174936 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:39.175667 (+ 731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:39.179016 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.179025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.179030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.179681 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.180131 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.180136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.180139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.180309 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.180883 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:39.180889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:39.181085 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.181246 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.181613 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":399,"threads_started":1}
I20260924 00:49:39.194118 2875 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39365
I20260924 00:49:39.194176 3012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39365 every 8 connection(s)
I20260924 00:49:39.195397 2875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:49:39.197778 3013 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45591
I20260924 00:49:39.198031 3013 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:39.198765 3013 heartbeater.cc:511] Master 127.0.222.62:45591 requested a full tablet report, sending...
I20260924 00:49:39.200309 2661 ts_manager.cc:194] Registered new tserver with Master: 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365)
I20260924 00:49:39.201133 2661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:43025
I20260924 00:49:39.203701 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 2875
I20260924 00:49:39.203840 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:49:39.204288 888 external_mini_cluster.cc:1428] Setting key 545d4385deac6dee74bbdaf47ae60b7c
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:49:39.246021 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45591
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:39.570194 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:39.570497 3017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:39.570600 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:39.578042 3017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:39.578161 3017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:39.578248 3017 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:49:39.578307 3017 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:49:39.578377 3017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:49:39.588572 3017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:39.590015 3017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:39.591298 3017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:39.603947 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:39.603937 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:39.603900 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:39.604640 3017 server_base.cc:1034] running on GCE node
I20260924 00:49:39.605288 3017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:39.606194 3017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:39.607403 3017 hybrid_clock.cc:648] HybridClock initialized: now 1790210979607343 us; error 49 us; skew 500 ppm
I20260924 00:49:39.607708 3017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:39 dist-test-slave-rtvz krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:39.611032 3017 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:49:39.612605 3017 webserver.cc:460] Webserver started at http://127.0.222.3:45419/ using document root <none> and password file <none>
I20260924 00:49:39.613322 3017 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:39.613453 3017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:39.613762 3017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:39.616734 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "dbb993b695b2428890c44c57805ffd8e"
format_stamp: "Formatted at 2026-09-24 00:49:39 on dist-test-slave-rtvz"
server_key: "eb9f1edaeda3c8b2b5aba6ef55b1625b"
server_key_iv: "6fd6fc7f02d011ce3a315d068ceaa74c"
server_key_version: "encryptionkey@0"
I20260924 00:49:39.617681 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "dbb993b695b2428890c44c57805ffd8e"
format_stamp: "Formatted at 2026-09-24 00:49:39 on dist-test-slave-rtvz"
server_key: "eb9f1edaeda3c8b2b5aba6ef55b1625b"
server_key_iv: "6fd6fc7f02d011ce3a315d068ceaa74c"
server_key_version: "encryptionkey@0"
I20260924 00:49:39.623020 3017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:49:39.626734 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:39.628293 3017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:39.628479 3017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "dbb993b695b2428890c44c57805ffd8e"
format_stamp: "Formatted at 2026-09-24 00:49:39 on dist-test-slave-rtvz"
server_key: "eb9f1edaeda3c8b2b5aba6ef55b1625b"
server_key_iv: "6fd6fc7f02d011ce3a315d068ceaa74c"
server_key_version: "encryptionkey@0"
I20260924 00:49:39.628687 3017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:39.667549 3017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:39.672151 3017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:39.672443 3017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:39.675237 3017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:39.675354 3017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:39.675474 3017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:39.675562 3017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:39.688613 2871 heartbeater.cc:503] Master 127.0.222.62:45591 was elected leader, sending a full tablet report...
Sep 24 00:49:39 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:39.699688 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.676098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:48859 (local address 127.0.222.62:45591)
0924 00:49:39.676292 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.676298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.678057 (+ 1759us) server_negotiation.cc:407] Connection header received
0924 00:49:39.681246 (+ 3189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.681255 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.681357 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.681456 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:39.684194 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.684979 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.687433 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.687966 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.688085 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.692549 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:39.692557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:39.692563 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:39.692609 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:39.696087 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.696884 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.696890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.696893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.697051 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.697648 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.697652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.697654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.697993 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:39.698146 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.699074 (+ 928us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.699318 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 00:49:39.701164 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.676746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45591 (local address 127.0.222.3:48859)
0924 00:49:39.677773 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.677878 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.680773 (+ 2895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.682097 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.682116 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.682976 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:39.683953 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.683992 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.685168 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.685174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.687273 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.687286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.688220 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.688229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.688579 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.689425 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:39.689455 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:39.692292 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:39.696245 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.696255 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.696275 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.696742 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.697159 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.697165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.697169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.697491 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.698120 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:39.698130 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:39.698598 (+ 468us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.698972 (+ 374us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.699591 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":423,"threads_started":1}
I20260924 00:49:39.719830 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.706658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35653 (local address 127.0.222.62:45591)
0924 00:49:39.706831 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.706837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.707563 (+ 726us) server_negotiation.cc:407] Connection header received
0924 00:49:39.707739 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.707744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.707812 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.707889 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:39.709028 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.709899 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.711347 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.711867 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.711991 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.714221 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:39.714226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:39.714229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:39.714261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:39.716967 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.717900 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.717906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.717909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.718072 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.718422 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.718427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.718430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.718711 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:39.718843 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.719253 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.719473 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:49:39.719944 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.706645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45591 (local address 127.0.222.3:35653)
0924 00:49:39.707464 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.707480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.707620 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.707933 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.707950 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.708611 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:39.708883 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.708892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.710035 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.710041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.711209 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.711224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.712116 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.712124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.712354 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.713075 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:39.713093 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:39.714037 (+ 944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:39.717104 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.717110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.717135 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.717790 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.718174 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.718179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.718182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.718324 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.718820 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:39.718825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:39.719005 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.719126 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.719469 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":464,"threads_started":1}
I20260924 00:49:39.728331 3017 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:35787
I20260924 00:49:39.728380 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:35787 every 8 connection(s)
I20260924 00:49:39.730299 3017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:49:39.733068 3154 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45591
I20260924 00:49:39.733332 3154 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:39.734278 3154 heartbeater.cc:511] Master 127.0.222.62:45591 requested a full tablet report, sending...
I20260924 00:49:39.734303 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 3017
I20260924 00:49:39.734460 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:49:39.734977 888 external_mini_cluster.cc:1428] Setting key c1b534f0c789e2989f818cc57f9b4871
I20260924 00:49:39.736308 2661 ts_manager.cc:194] Registered new tserver with Master: dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787)
I20260924 00:49:39.737286 2661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:48859
I20260924 00:49:39.738592 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:49:39 dist-test-slave-rtvz krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:49:39 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:39.791260 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.777603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.0.222.62:45591)
0924 00:49:39.777759 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.777764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.778149 (+ 385us) server_negotiation.cc:407] Connection header received
0924 00:49:39.778312 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.778316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.778382 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.778501 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:39.779795 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.780516 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.781982 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.782449 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.782559 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.786338 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:39.786343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:39.786346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:39.786381 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:39.788965 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.789689 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.789693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.789695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.789816 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.790128 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.790131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.790133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.790406 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:39.790580 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.790815 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.790986 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:49:39.795471 2660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:49:39.802371 2660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:49:39.825907 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.815618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.0.1:45608)
0924 00:49:39.816750 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.816793 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.816998 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.819210 (+ 2212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.819217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.819247 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:39.819699 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.819713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.821510 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.821515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.823461 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.823476 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.824802 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.824808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.825005 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.825064 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.825184 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.825442 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"spinlock_wait_cycles":15744,"thread_start_us":1668,"threads_started":1}
I20260924 00:49:39.825953 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.815711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.0.222.1:41785)
0924 00:49:39.818540 (+ 2829us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.818549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.818578 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:49:39.818677 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.818686 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.818967 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.819941 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:39.820154 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.821392 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.823625 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.824609 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.824730 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.824973 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.825490 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.825675 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2667,"thread_start_us":2550,"threads_started":1}
I20260924 00:49:39.827205 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.815726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35787 (local address 127.0.0.1:40778)
0924 00:49:39.817577 (+ 1851us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.817594 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.817689 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.819666 (+ 1977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.819673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.819696 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:39.820088 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.820102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.822233 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.822238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.823777 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.823789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.826468 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.826474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.826638 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.826671 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.826733 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.826856 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"mutex_wait_us":1,"thread_start_us":647,"threads_started":1}
I20260924 00:49:39.828313 3170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.816160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:39365 (local address 127.0.0.1:39428)
0924 00:49:39.819215 (+ 3055us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.819231 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.819455 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.820049 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.820053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.820078 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:39.820457 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.820469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.822170 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.822176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.825708 (+ 3532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.825719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.827607 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.827613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.827739 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.827760 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.827834 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.827963 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2940,"spinlock_wait_cycles":896,"thread_start_us":2806,"threads_started":1}
I20260924 00:49:39.828336 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.816443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.0.222.2:39365)
0924 00:49:39.818130 (+ 1687us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.818140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.819315 (+ 1175us) server_negotiation.cc:407] Connection header received
0924 00:49:39.819610 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.819618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.819843 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.820056 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:39.820631 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.821996 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.825898 (+ 3902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.827228 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.827329 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.827551 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.827905 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.828063 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1549,"thread_start_us":1433,"threads_started":1}
I20260924 00:49:39.828385 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.816247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.222.3:35787)
0924 00:49:39.818988 (+ 2741us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.818996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.819027 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:49:39.819202 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.819216 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.819479 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.819674 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:39.820298 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.822103 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.823932 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.826243 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.827562 (+ 1319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.827797 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.827876 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.828037 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2557,"thread_start_us":2382,"threads_started":1}
I20260924 00:49:39.830302 2800 tablet_service.cc:1467] Processing CreateTablet for tablet 6c45d8a615ec449f8bf0369614cd51b9 (DEFAULT_TABLE table=test-table [id=b1de849610be40f998ba6fa3c4205905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:39.831557 2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c45d8a615ec449f8bf0369614cd51b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:39.831835 3083 tablet_service.cc:1467] Processing CreateTablet for tablet 6c45d8a615ec449f8bf0369614cd51b9 (DEFAULT_TABLE table=test-table [id=b1de849610be40f998ba6fa3c4205905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:39.831837 2942 tablet_service.cc:1467] Processing CreateTablet for tablet 6c45d8a615ec449f8bf0369614cd51b9 (DEFAULT_TABLE table=test-table [id=b1de849610be40f998ba6fa3c4205905]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:39.833164 2942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c45d8a615ec449f8bf0369614cd51b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:39.833163 3083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c45d8a615ec449f8bf0369614cd51b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:39.844590 3173 tablet_bootstrap.cc:492] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Bootstrap starting.
I20260924 00:49:39.847702 3175 tablet_bootstrap.cc:492] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Bootstrap starting.
I20260924 00:49:39.848431 3173 tablet_bootstrap.cc:654] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:39.848500 3174 tablet_bootstrap.cc:492] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Bootstrap starting.
I20260924 00:49:39.849938 3173 log.cc:824] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:39.852038 3174 tablet_bootstrap.cc:654] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:39.852805 3175 tablet_bootstrap.cc:654] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:39.853398 3174 log.cc:824] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:39.853525 3173 tablet_bootstrap.cc:492] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: No bootstrap required, opened a new log
I20260924 00:49:39.853891 3173 ts_tablet_manager.cc:1397] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 00:49:39.853994 3175 log.cc:824] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:39.856689 3174 tablet_bootstrap.cc:492] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: No bootstrap required, opened a new log
I20260924 00:49:39.856974 3174 ts_tablet_manager.cc:1397] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 00:49:39.858836 3175 tablet_bootstrap.cc:492] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: No bootstrap required, opened a new log
I20260924 00:49:39.859056 3175 ts_tablet_manager.cc:1397] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 00:49:39.864509 3173 raft_consensus.cc:348] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.865020 3173 raft_consensus.cc:374] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:39.865185 3173 raft_consensus.cc:729] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac3deb94f5d48a1b5106e76eef69daa, State: Initialized, Role: FOLLOWER
I20260924 00:49:39.866025 3173 consensus_queue.cc:260] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.866202 3175 raft_consensus.cc:348] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.866627 3175 raft_consensus.cc:374] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:39.866747 3175 raft_consensus.cc:729] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb993b695b2428890c44c57805ffd8e, State: Initialized, Role: FOLLOWER
I20260924 00:49:39.867352 3175 consensus_queue.cc:260] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.867210 3174 raft_consensus.cc:348] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.867697 3174 raft_consensus.cc:374] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:39.867842 3174 raft_consensus.cc:729] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534411c11bc44528aad2fe49b731f91c, State: Initialized, Role: FOLLOWER
I20260924 00:49:39.868331 3154 heartbeater.cc:503] Master 127.0.222.62:45591 was elected leader, sending a full tablet report...
I20260924 00:49:39.868590 3174 consensus_queue.cc:260] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.869930 3013 heartbeater.cc:503] Master 127.0.222.62:45591 was elected leader, sending a full tablet report...
I20260924 00:49:39.870393 3174 ts_tablet_manager.cc:1428] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 00:49:39.870653 3175 ts_tablet_manager.cc:1428] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 00:49:39.872270 3173 ts_tablet_manager.cc:1428] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260924 00:49:39.889153 3179 raft_consensus.cc:482] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:39.889500 3179 raft_consensus.cc:504] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.891345 3179 leader_election.cc:288] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787), 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365)
I20260924 00:49:39.897974 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.891763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:38491 (local address 127.0.222.3:35787)
0924 00:49:39.892021 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.892029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.892698 (+ 669us) server_negotiation.cc:407] Connection header received
0924 00:49:39.892888 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.892894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.892969 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.893051 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:39.893733 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.894480 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.896253 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.897006 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.897107 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.897234 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.897616 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.897734 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 00:49:39.897974 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.891835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:32941 (local address 127.0.222.2:39365)
0924 00:49:39.892045 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.892051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.892802 (+ 751us) server_negotiation.cc:407] Connection header received
0924 00:49:39.893062 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.893068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.893142 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.893262 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:39.893856 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.894586 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.896276 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.897015 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.897138 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.897272 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.897594 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.897720 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:49:39.898205 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.891594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35787 (local address 127.0.222.1:38491)
0924 00:49:39.892594 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.892609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.892743 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.893075 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.893080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.893105 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:39.893609 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.893622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.894609 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.894614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.896110 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.896121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.897251 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.897259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.897455 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.897495 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.897578 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.897718 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":33,"spinlock_wait_cycles":21248,"thread_start_us":689,"threads_started":1}
I20260924 00:49:39.898314 3182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.891600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:39365 (local address 127.0.222.1:32941)
0924 00:49:39.892708 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.892726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.892904 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.893290 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.893296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.893335 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:39.893715 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.893726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.894710 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.894714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.896130 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.896141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.897289 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.897294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.897442 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.897475 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.897545 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.897666 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":45,"spinlock_wait_cycles":1920,"thread_start_us":630,"threads_started":1}
I20260924 00:49:39.899258 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c45d8a615ec449f8bf0369614cd51b9" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb993b695b2428890c44c57805ffd8e" is_pre_election: true
I20260924 00:49:39.899260 2962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c45d8a615ec449f8bf0369614cd51b9" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c" is_pre_election: true
I20260924 00:49:39.899830 3108 raft_consensus.cc:2454] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 0.
I20260924 00:49:39.899824 2962 raft_consensus.cc:2454] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 0.
I20260924 00:49:39.900492 2752 leader_election.cc:302] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac3deb94f5d48a1b5106e76eef69daa, dbb993b695b2428890c44c57805ffd8e; no voters:
I20260924 00:49:39.900895 3179 raft_consensus.cc:2781] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:39.901028 3179 raft_consensus.cc:482] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:39.901127 3179 raft_consensus.cc:3037] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:39.903318 3179 raft_consensus.cc:504] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.903892 3179 leader_election.cc:288] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 election: Requested vote from peers dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787), 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365)
I20260924 00:49:39.904402 2962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c45d8a615ec449f8bf0369614cd51b9" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c"
I20260924 00:49:39.904441 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c45d8a615ec449f8bf0369614cd51b9" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb993b695b2428890c44c57805ffd8e"
I20260924 00:49:39.904592 2962 raft_consensus.cc:3037] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:39.904645 3108 raft_consensus.cc:3037] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:39.907259 2962 raft_consensus.cc:2454] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 1.
I20260924 00:49:39.907413 3108 raft_consensus.cc:2454] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 1.
I20260924 00:49:39.907707 2749 leader_election.cc:302] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac3deb94f5d48a1b5106e76eef69daa, 534411c11bc44528aad2fe49b731f91c; no voters:
I20260924 00:49:39.908098 3179 raft_consensus.cc:2781] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:39.908641 3179 raft_consensus.cc:686] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 LEADER]: Becoming Leader. State: Replica: 3ac3deb94f5d48a1b5106e76eef69daa, State: Running, Role: LEADER
I20260924 00:49:39.909144 3179 consensus_queue.cc:237] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } }
I20260924 00:49:39.914197 2661 catalog_manager.cc:5495] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1). New cstate: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } health_report { overall_health: UNKNOWN } } }
W20260924 00:49:39.930976 2872 tablet.cc:2369] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:49:39.947362 3014 tablet.cc:2369] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 00:49:39 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:39.975008 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.955895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45591 (local address 127.0.0.1:47156)
0924 00:49:39.956608 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:39.956637 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:39.956810 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:39.957261 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:39.957267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:39.958168 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:39.958496 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.958507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.959581 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.959585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.960664 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:39.960677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.961440 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.961446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:39.961572 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.967156 (+ 5584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:39.967178 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:39.969605 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:39.972477 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.972490 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.972497 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.972928 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.973286 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:39.973290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:39.973292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:39.973425 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:39.974128 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:39.974135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:39.974298 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 00:49:39.974430 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 00:49:39.974658 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":393,"threads_started":1}
I20260924 00:49:39.975178 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:39.955783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.0.222.62:45591)
0924 00:49:39.955988 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:39.955995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:39.956760 (+ 765us) server_negotiation.cc:407] Connection header received
0924 00:49:39.956923 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:39.956927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:39.956993 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:39.957083 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:39.958671 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.959423 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.960817 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:39.961263 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:39.961363 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:39.969836 (+ 8473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:39.969844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:39.969848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:39.969892 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:39.972325 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.973038 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.973043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.973045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.973172 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:39.973626 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:39.973631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:39.973633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:39.973998 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:39.974126 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:39.974560 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 00:49:39.974803 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":67}
I20260924 00:49:39.980476 2661 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:47156:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:49:39.981693 2661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 00:49:39.983337 3155 tablet.cc:2369] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:49:39.989415 2942 tablet_service.cc:1467] Processing CreateTablet for tablet 83ac8ca2edd548739f0279ff696bbb99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ea1ee11acca433598cd0cbe56dbb958]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:49:39.989133 2800 tablet_service.cc:1467] Processing CreateTablet for tablet 83ac8ca2edd548739f0279ff696bbb99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ea1ee11acca433598cd0cbe56dbb958]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:49:39.989727 3083 tablet_service.cc:1467] Processing CreateTablet for tablet 83ac8ca2edd548739f0279ff696bbb99 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ea1ee11acca433598cd0cbe56dbb958]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:49:39.990250 2942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ac8ca2edd548739f0279ff696bbb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:39.990269 2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ac8ca2edd548739f0279ff696bbb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:39.990561 3083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ac8ca2edd548739f0279ff696bbb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:39.995937 3174 tablet_bootstrap.cc:492] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: Bootstrap starting.
I20260924 00:49:39.997426 3174 tablet_bootstrap.cc:654] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:39.997476 3175 tablet_bootstrap.cc:492] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: Bootstrap starting.
I20260924 00:49:39.999048 3175 tablet_bootstrap.cc:654] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:39.999537 3174 tablet_bootstrap.cc:492] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: No bootstrap required, opened a new log
I20260924 00:49:39.999663 3174 ts_tablet_manager.cc:1397] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 00:49:39.999945 3173 tablet_bootstrap.cc:492] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: Bootstrap starting.
I20260924 00:49:40.000332 3174 raft_consensus.cc:348] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.000553 3174 raft_consensus.cc:374] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:40.000691 3174 raft_consensus.cc:729] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534411c11bc44528aad2fe49b731f91c, State: Initialized, Role: FOLLOWER
I20260924 00:49:40.000988 3174 consensus_queue.cc:260] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.001474 3173 tablet_bootstrap.cc:654] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:40.001431 3181 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.001767 3181 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:40.002162 3174 ts_tablet_manager.cc:1428] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 00:49:40.004150 3175 tablet_bootstrap.cc:492] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: No bootstrap required, opened a new log
I20260924 00:49:40.004355 3175 ts_tablet_manager.cc:1397] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 00:49:40.005347 3175 raft_consensus.cc:348] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.005600 3175 raft_consensus.cc:374] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:40.005690 3173 tablet_bootstrap.cc:492] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: No bootstrap required, opened a new log
I20260924 00:49:40.005700 3175 raft_consensus.cc:729] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb993b695b2428890c44c57805ffd8e, State: Initialized, Role: FOLLOWER
I20260924 00:49:40.005842 3173 ts_tablet_manager.cc:1397] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260924 00:49:40.005923 3175 consensus_queue.cc:260] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.006582 3175 ts_tablet_manager.cc:1428] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:49:40.006481 3180 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.006628 3173 raft_consensus.cc:348] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.006865 3173 raft_consensus.cc:374] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:40.006953 3180 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:40.006973 3173 raft_consensus.cc:729] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac3deb94f5d48a1b5106e76eef69daa, State: Initialized, Role: FOLLOWER
I20260924 00:49:40.007231 3173 consensus_queue.cc:260] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.007702 3179 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.007908 3173 ts_tablet_manager.cc:1428] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:49:40.008018 3179 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:40.250979 3179 raft_consensus.cc:482] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:40.251212 3179 raft_consensus.cc:504] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.251880 3179 leader_election.cc:288] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365), dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787)
I20260924 00:49:40.252455 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ac8ca2edd548739f0279ff696bbb99" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb993b695b2428890c44c57805ffd8e" is_pre_election: true
I20260924 00:49:40.252452 2962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ac8ca2edd548739f0279ff696bbb99" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c" is_pre_election: true
I20260924 00:49:40.252681 3108 raft_consensus.cc:2454] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 0.
I20260924 00:49:40.252717 2962 raft_consensus.cc:2454] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 0.
I20260924 00:49:40.253067 2752 leader_election.cc:302] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac3deb94f5d48a1b5106e76eef69daa, dbb993b695b2428890c44c57805ffd8e; no voters:
I20260924 00:49:40.253371 3179 raft_consensus.cc:2781] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:40.253470 3179 raft_consensus.cc:482] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:40.253593 3179 raft_consensus.cc:3037] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:40.255694 3179 raft_consensus.cc:504] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.256263 3179 leader_election.cc:288] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 election: Requested vote from peers 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365), dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787)
I20260924 00:49:40.256781 2962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ac8ca2edd548739f0279ff696bbb99" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c"
I20260924 00:49:40.256795 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ac8ca2edd548739f0279ff696bbb99" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb993b695b2428890c44c57805ffd8e"
I20260924 00:49:40.256997 2962 raft_consensus.cc:3037] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:40.257085 3108 raft_consensus.cc:3037] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:40.259027 2962 raft_consensus.cc:2454] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 1.
I20260924 00:49:40.259050 3108 raft_consensus.cc:2454] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 1.
I20260924 00:49:40.259438 2749 leader_election.cc:302] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac3deb94f5d48a1b5106e76eef69daa, 534411c11bc44528aad2fe49b731f91c; no voters:
I20260924 00:49:40.259745 3179 raft_consensus.cc:2781] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:40.259917 3179 raft_consensus.cc:686] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 LEADER]: Becoming Leader. State: Replica: 3ac3deb94f5d48a1b5106e76eef69daa, State: Running, Role: LEADER
I20260924 00:49:40.260150 3179 consensus_queue.cc:237] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.261772 3184 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: TxnStatusTablet state changed. Reason: New leader 3ac3deb94f5d48a1b5106e76eef69daa. Latest consensus state: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.262236 3184 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:49:40.263175 2661 catalog_manager.cc:5495] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1). New cstate: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } health_report { overall_health: UNKNOWN } } }
I20260924 00:49:40.264662 3196 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:49:40.265028 3196 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:49:40.290319 3184 consensus_queue.cc:1035] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:40.306840 3179 consensus_queue.cc:1035] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:40.312943 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.302134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.0.222.1:41785)
0924 00:49:40.302855 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:40.302861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:40.302879 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:49:40.302979 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:40.302985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:40.303126 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:40.303272 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:40.304070 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.305351 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.307578 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.308223 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.308346 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.311381 (+ 3035us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.312110 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.312234 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:40.312638 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 00:49:40.312721 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579}
I20260924 00:49:40.314275 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.302087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.0.1:45612)
0924 00:49:40.302389 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:40.302411 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:40.302558 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:40.303340 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:40.303346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:40.303362 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:49:40.303599 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.303610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.305593 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.305600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.307379 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.307396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.308404 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.308410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.311016 (+ 2606us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.311117 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.312511 (+ 1394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.312534 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 00:49:40.312617 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:49:40.312821 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 00:49:40.323490 2962 raft_consensus.cc:1261] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Refusing update from remote peer 3ac3deb94f5d48a1b5106e76eef69daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:40.323589 3108 raft_consensus.cc:1261] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Refusing update from remote peer 3ac3deb94f5d48a1b5106e76eef69daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:40.324537 3179 consensus_queue.cc:1035] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:40.325431 3184 consensus_queue.cc:1035] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:40.329583 3181 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: TxnStatusTablet state changed. Reason: New leader 3ac3deb94f5d48a1b5106e76eef69daa. Latest consensus state: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.329876 3181 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:40.331542 3180 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: TxnStatusTablet state changed. Reason: New leader 3ac3deb94f5d48a1b5106e76eef69daa. Latest consensus state: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.331830 3180 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:40.336275 3180 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.336566 3180 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P dbb993b695b2428890c44c57805ffd8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:40.339073 3201 mvcc.cc:205] Tried to move back new op lower bound from 7332704175395004416 to 7332704175147745280. Current Snapshot: MvccSnapshot[applied={T|T < 7332704175395004416 or (T in {7332704175395004416})}]
I20260924 00:49:40.342052 3179 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.342409 3179 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:49:40.346889 3206 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.347273 3206 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P 3ac3deb94f5d48a1b5106e76eef69daa: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:49:40.347707 3203 mvcc.cc:205] Tried to move back new op lower bound from 7332704175395004416 to 7332704175147745280. Current Snapshot: MvccSnapshot[applied={T|T < 7332704175395004416}]
I20260924 00:49:40.348724 3181 tablet_replica.cc:439] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } } }
I20260924 00:49:40.348968 3181 tablet_replica.cc:442] T 83ac8ca2edd548739f0279ff696bbb99 P 534411c11bc44528aad2fe49b731f91c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:49:40.358584 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.352446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.0.222.1:41785)
0924 00:49:40.352629 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:40.352635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:40.352653 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:49:40.352851 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:40.352857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:40.352941 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:40.353032 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:40.353574 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.354761 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.356427 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.357084 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.357232 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.357775 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.357929 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.358020 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:40.358155 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 00:49:40.358243 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 00:49:40.376767 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.370222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:37191 (local address 127.0.222.1:41785)
0924 00:49:40.370439 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:40.370449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:40.371095 (+ 646us) server_negotiation.cc:407] Connection header received
0924 00:49:40.371252 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:40.371258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:40.371340 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:40.371431 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:40.371905 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.373011 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.374603 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.375291 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.375408 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.375975 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.376123 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.376209 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:40.376318 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 00:49:40.376429 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
I20260924 00:49:40.376791 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.370188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.222.1:37191)
0924 00:49:40.371002 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:40.371017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:40.371146 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:40.371460 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:40.371464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:40.371474 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:49:40.371770 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.371778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.373139 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.373143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.374467 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.374484 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.375491 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.375498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.375689 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.375767 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.376222 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.376230 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 00:49:40.376284 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 00:49:40.376441 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":38,"thread_start_us":503,"threads_started":1}
I20260924 00:49:40.443780 2660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:40.444041 2660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:40.447639 2660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: Sending DeleteTablet for 3 replicas of tablet 6c45d8a615ec449f8bf0369614cd51b9
I20260924 00:49:40.448731 2799 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c45d8a615ec449f8bf0369614cd51b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:40 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:45608
I20260924 00:49:40.449029 2942 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c45d8a615ec449f8bf0369614cd51b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:40 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:39428
I20260924 00:49:40.449546 3083 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c45d8a615ec449f8bf0369614cd51b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:40 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40778
I20260924 00:49:40.449909 3217 tablet_replica.cc:330] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: stopping tablet replica
I20260924 00:49:40.450409 3217 raft_consensus.cc:2229] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:40.450858 3217 raft_consensus.cc:2258] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:40.450940 2660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:49:40.451629 2660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:49:40.453362 3217 ts_tablet_manager.cc:1905] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:40.454592 3218 tablet_replica.cc:330] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: stopping tablet replica
I20260924 00:49:40.455015 3218 raft_consensus.cc:2229] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:40.455369 3218 raft_consensus.cc:2258] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:40.457360 3218 ts_tablet_manager.cc:1905] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:40.459930 2942 tablet_service.cc:1467] Processing CreateTablet for tablet 027e4d0a55b64f4fae45ed53cb311e2f (DEFAULT_TABLE table=test-table [id=494f2aa33f584f51b1b33c97e2278572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:40.460430 2942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027e4d0a55b64f4fae45ed53cb311e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:40.464473 3218 ts_tablet_manager.cc:1918] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:40.464672 3218 log.cc:1196] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/6c45d8a615ec449f8bf0369614cd51b9
I20260924 00:49:40.465279 3218 ts_tablet_manager.cc:1939] T 6c45d8a615ec449f8bf0369614cd51b9 P dbb993b695b2428890c44c57805ffd8e: Deleting consensus metadata
I20260924 00:49:40.465675 3216 tablet_replica.cc:330] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: stopping tablet replica
I20260924 00:49:40.465656 3083 tablet_service.cc:1467] Processing CreateTablet for tablet 027e4d0a55b64f4fae45ed53cb311e2f (DEFAULT_TABLE table=test-table [id=494f2aa33f584f51b1b33c97e2278572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:40.466060 3217 ts_tablet_manager.cc:1918] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:40.466157 3216 raft_consensus.cc:2229] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:49:40.466325 3217 log.cc:1196] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/6c45d8a615ec449f8bf0369614cd51b9
I20260924 00:49:40.466382 3083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027e4d0a55b64f4fae45ed53cb311e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:40.466811 3216 raft_consensus.cc:2258] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:40.466967 3217 ts_tablet_manager.cc:1939] T 6c45d8a615ec449f8bf0369614cd51b9 P 534411c11bc44528aad2fe49b731f91c: Deleting consensus metadata
I20260924 00:49:40.467327 2648 catalog_manager.cc:4841] TS dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787): tablet 6c45d8a615ec449f8bf0369614cd51b9 (table test-table [id=b1de849610be40f998ba6fa3c4205905]) successfully deleted
I20260924 00:49:40.468662 2800 tablet_service.cc:1467] Processing CreateTablet for tablet 027e4d0a55b64f4fae45ed53cb311e2f (DEFAULT_TABLE table=test-table [id=494f2aa33f584f51b1b33c97e2278572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:40.468751 2645 catalog_manager.cc:4841] TS 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365): tablet 6c45d8a615ec449f8bf0369614cd51b9 (table test-table [id=b1de849610be40f998ba6fa3c4205905]) successfully deleted
I20260924 00:49:40.468842 3216 ts_tablet_manager.cc:1905] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:40.469314 2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027e4d0a55b64f4fae45ed53cb311e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:40.477031 3216 ts_tablet_manager.cc:1918] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:40.477257 3216 log.cc:1196] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/6c45d8a615ec449f8bf0369614cd51b9
I20260924 00:49:40.477360 3173 tablet_bootstrap.cc:492] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: Bootstrap starting.
I20260924 00:49:40.477916 3216 ts_tablet_manager.cc:1939] T 6c45d8a615ec449f8bf0369614cd51b9 P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting consensus metadata
I20260924 00:49:40.479409 3173 tablet_bootstrap.cc:654] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:40.479882 2647 catalog_manager.cc:4841] TS 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785): tablet 6c45d8a615ec449f8bf0369614cd51b9 (table test-table [id=b1de849610be40f998ba6fa3c4205905]) successfully deleted
I20260924 00:49:40.480262 3174 tablet_bootstrap.cc:492] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: Bootstrap starting.
I20260924 00:49:40.481971 3174 tablet_bootstrap.cc:654] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:40.482159 3175 tablet_bootstrap.cc:492] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: Bootstrap starting.
I20260924 00:49:40.484097 3175 tablet_bootstrap.cc:654] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:40.484712 3174 tablet_bootstrap.cc:492] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: No bootstrap required, opened a new log
I20260924 00:49:40.484722 3173 tablet_bootstrap.cc:492] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: No bootstrap required, opened a new log
I20260924 00:49:40.484910 3174 ts_tablet_manager.cc:1397] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 00:49:40.484923 3173 ts_tablet_manager.cc:1397] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 00:49:40.485771 3174 raft_consensus.cc:348] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.485778 3173 raft_consensus.cc:348] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.486043 3174 raft_consensus.cc:374] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:40.486119 3174 raft_consensus.cc:729] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534411c11bc44528aad2fe49b731f91c, State: Initialized, Role: FOLLOWER
I20260924 00:49:40.486269 3173 raft_consensus.cc:374] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:40.486438 3173 raft_consensus.cc:729] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac3deb94f5d48a1b5106e76eef69daa, State: Initialized, Role: FOLLOWER
I20260924 00:49:40.486433 3174 consensus_queue.cc:260] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.487026 3174 ts_tablet_manager.cc:1428] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 00:49:40.486860 3173 consensus_queue.cc:260] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.488669 3173 ts_tablet_manager.cc:1428] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 00:49:40.493811 3175 tablet_bootstrap.cc:492] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: No bootstrap required, opened a new log
I20260924 00:49:40.494002 3175 ts_tablet_manager.cc:1397] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 00:49:40.494807 3175 raft_consensus.cc:348] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.495010 3175 raft_consensus.cc:374] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:40.495087 3175 raft_consensus.cc:729] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb993b695b2428890c44c57805ffd8e, State: Initialized, Role: FOLLOWER
I20260924 00:49:40.495297 3175 consensus_queue.cc:260] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.495769 3175 ts_tablet_manager.cc:1428] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 00:49:40.678122 3180 raft_consensus.cc:482] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:40.678391 3180 raft_consensus.cc:504] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.680317 3180 leader_election.cc:288] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785), 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365)
I20260924 00:49:40.701776 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.681004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:40993 (local address 127.0.222.1:41785)
0924 00:49:40.681183 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:40.681189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:40.692943 (+ 11754us) server_negotiation.cc:407] Connection header received
0924 00:49:40.693328 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:40.693349 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:40.693436 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:40.693506 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:40.694285 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.695460 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.698736 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.700226 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.700359 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.700475 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:40.701191 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 00:49:40.701463 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:49:40.701802 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.681120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:38219 (local address 127.0.222.2:39365)
0924 00:49:40.692371 (+ 11251us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:40.692378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:40.693361 (+ 983us) server_negotiation.cc:407] Connection header received
0924 00:49:40.693734 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:40.693739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:40.693818 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:40.693887 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:49:40.694711 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.695844 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.698732 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.700099 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.700235 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.700357 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:40.701321 (+ 964us) server_negotiation.cc:299] Negotiation successful
0924 00:49:40.701471 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11088,"thread_start_us":10970,"threads_started":1}
I20260924 00:49:40.702620 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.680772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:39365 (local address 127.0.222.3:38219)
0924 00:49:40.693233 (+ 12461us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:40.693254 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:40.693483 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:40.694141 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:40.694146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:40.694175 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:40.694482 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.694492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.696041 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.696046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.698475 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.698499 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.700892 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.700898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.701076 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.701108 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:49:40.702006 (+ 898us) client_negotiation.cc:240] Negotiation successful
0924 00:49:40.702169 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12275,"mutex_wait_us":126,"thread_start_us":12050,"threads_started":1}
I20260924 00:49:40.703078 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.680597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.222.3:40993)
0924 00:49:40.692779 (+ 12182us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:40.692808 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:40.693083 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:40.693800 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:40.693805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:40.693835 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:49:40.694090 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.694102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.695672 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.695678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.698475 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.698499 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.700662 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.700669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.700964 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.701000 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 00:49:40.702379 (+ 1379us) client_negotiation.cc:240] Negotiation successful
0924 00:49:40.702541 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11993,"mutex_wait_us":81,"thread_start_us":11845,"threads_started":1}
I20260924 00:49:40.703438 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027e4d0a55b64f4fae45ed53cb311e2f" candidate_uuid: "dbb993b695b2428890c44c57805ffd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c" is_pre_election: true
I20260924 00:49:40.703784 2961 raft_consensus.cc:2454] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb993b695b2428890c44c57805ffd8e in term 0.
I20260924 00:49:40.704423 3033 leader_election.cc:302] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 534411c11bc44528aad2fe49b731f91c, dbb993b695b2428890c44c57805ffd8e; no voters:
I20260924 00:49:40.704432 2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027e4d0a55b64f4fae45ed53cb311e2f" candidate_uuid: "dbb993b695b2428890c44c57805ffd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" is_pre_election: true
I20260924 00:49:40.704916 3180 raft_consensus.cc:2781] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:40.704931 2820 raft_consensus.cc:2454] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb993b695b2428890c44c57805ffd8e in term 0.
I20260924 00:49:40.705128 3180 raft_consensus.cc:482] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:40.705291 3180 raft_consensus.cc:3037] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:40.708680 3180 raft_consensus.cc:504] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.710026 2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027e4d0a55b64f4fae45ed53cb311e2f" candidate_uuid: "dbb993b695b2428890c44c57805ffd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac3deb94f5d48a1b5106e76eef69daa"
I20260924 00:49:40.710258 2820 raft_consensus.cc:3037] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:40.710904 3180 leader_election.cc:288] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 election: Requested vote from peers 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785), 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365)
I20260924 00:49:40.711018 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027e4d0a55b64f4fae45ed53cb311e2f" candidate_uuid: "dbb993b695b2428890c44c57805ffd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c"
I20260924 00:49:40.711244 2961 raft_consensus.cc:3037] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:40.713783 2820 raft_consensus.cc:2454] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb993b695b2428890c44c57805ffd8e in term 1.
I20260924 00:49:40.714310 2961 raft_consensus.cc:2454] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb993b695b2428890c44c57805ffd8e in term 1.
I20260924 00:49:40.714411 3035 leader_election.cc:302] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac3deb94f5d48a1b5106e76eef69daa, dbb993b695b2428890c44c57805ffd8e; no voters:
I20260924 00:49:40.714793 3180 raft_consensus.cc:2781] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:40.715353 3180 raft_consensus.cc:686] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 1 LEADER]: Becoming Leader. State: Replica: dbb993b695b2428890c44c57805ffd8e, State: Running, Role: LEADER
I20260924 00:49:40.715864 3180 consensus_queue.cc:237] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:40.720198 2659 catalog_manager.cc:5495] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbb993b695b2428890c44c57805ffd8e (127.0.222.3). New cstate: current_term: 1 leader_uuid: "dbb993b695b2428890c44c57805ffd8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } health_report { overall_health: HEALTHY } } }
I20260924 00:49:40.789330 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.781968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.0.222.3:35787)
0924 00:49:40.782673 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:40.782677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:40.782696 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:49:40.782784 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:40.782787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:40.782866 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:40.783013 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:40.783492 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.784349 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.786320 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.786906 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.787022 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.787569 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.788441 (+ 872us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.788597 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:40.788749 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:49:40.788840 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":484,"threads_started":1}
I20260924 00:49:40.806492 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.800439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:45309 (local address 127.0.222.1:41785)
0924 00:49:40.800642 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:40.800648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:40.801244 (+ 596us) server_negotiation.cc:407] Connection header received
0924 00:49:40.801437 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:40.801441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:40.801515 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:40.801592 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:40.802290 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.803231 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.804620 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.805081 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.805223 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.805671 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.805800 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.805892 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:40.806023 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 00:49:40.806108 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:49:40.806511 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.800370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.222.3:45309)
0924 00:49:40.801102 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:40.801117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:40.801294 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:40.801634 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:40.801639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:40.801651 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:49:40.802037 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.802055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.803406 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.803411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.804476 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.804489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.805291 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.805297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.805434 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.805499 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.805903 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.805915 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 00:49:40.805985 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 00:49:40.806138 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":361,"threads_started":1}
I20260924 00:49:40.816871 2961 raft_consensus.cc:1261] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Refusing update from remote peer dbb993b695b2428890c44c57805ffd8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:40.817499 2820 raft_consensus.cc:1261] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Refusing update from remote peer dbb993b695b2428890c44c57805ffd8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:40.817710 3225 consensus_queue.cc:1035] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:40.818163 3180 consensus_queue.cc:1035] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:40.835623 3200 mvcc.cc:205] Tried to move back new op lower bound from 7332704177419268096 to 7332704177017094144. Current Snapshot: MvccSnapshot[applied={T|T < 7332704177419268096}]
I20260924 00:49:40.861590 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.856679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55401 (local address 127.0.222.3:35787)
0924 00:49:40.856832 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:40.856836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:40.856850 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:49:40.856899 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:40.856903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:40.856970 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:40.857043 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:40.857428 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.858146 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.859820 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.860379 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.860490 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.860867 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.860972 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.861044 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:40.861180 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 00:49:40.861289 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:49:40.861606 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:40.856179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:35787 (local address 127.0.222.1:55401)
0924 00:49:40.856364 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:40.856383 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:40.856493 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:40.857083 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:40.857089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:40.857101 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:49:40.857307 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.857316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.858319 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.858324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.859704 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:40.859719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:40.860515 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:40.860521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:40.860683 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:40.860721 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:49:40.861067 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:49:40.861080 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 00:49:40.861150 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 00:49:40.861318 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 00:49:40.862885 3201 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 00:49:42.464232 2659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:42.464639 2659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:42.470551 2659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: Sending DeleteTablet for 3 replicas of tablet 027e4d0a55b64f4fae45ed53cb311e2f
I20260924 00:49:42.471597 2942 tablet_service.cc:1514] Processing DeleteTablet for tablet 027e4d0a55b64f4fae45ed53cb311e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:42 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:39428
I20260924 00:49:42.471608 2800 tablet_service.cc:1514] Processing DeleteTablet for tablet 027e4d0a55b64f4fae45ed53cb311e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:42 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:45608
I20260924 00:49:42.471690 3083 tablet_service.cc:1514] Processing DeleteTablet for tablet 027e4d0a55b64f4fae45ed53cb311e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:42 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40778
I20260924 00:49:42.472851 3261 tablet_replica.cc:330] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: stopping tablet replica
I20260924 00:49:42.473133 3260 tablet_replica.cc:330] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: stopping tablet replica
I20260924 00:49:42.473172 3261 raft_consensus.cc:2229] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:42.473496 3260 raft_consensus.cc:2229] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:42.473652 3261 raft_consensus.cc:2258] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:42.474056 3260 raft_consensus.cc:2258] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:42.474956 3261 ts_tablet_manager.cc:1905] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:42.475134 3260 ts_tablet_manager.cc:1905] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:42.479825 3262 tablet_replica.cc:330] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: stopping tablet replica
I20260924 00:49:42.480152 3262 raft_consensus.cc:2229] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:49:42.480911 3262 raft_consensus.cc:2258] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:42.482067 3262 ts_tablet_manager.cc:1905] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 00:49:42 dist-test-slave-rtvz krb5kdc[2616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 00:49:42.589128 888 mini_kdc.cc:317] Time spent kinit for test-user: real 0.115s user 0.000s sys 0.011s
I20260924 00:49:42.589188 3260 ts_tablet_manager.cc:1918] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:49:42.589190 3261 ts_tablet_manager.cc:1918] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:49:42.589387 3261 log.cc:1196] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/027e4d0a55b64f4fae45ed53cb311e2f
I20260924 00:49:42.589435 3260 log.cc:1196] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/027e4d0a55b64f4fae45ed53cb311e2f
I20260924 00:49:42.589843 3261 ts_tablet_manager.cc:1939] T 027e4d0a55b64f4fae45ed53cb311e2f P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting consensus metadata
I20260924 00:49:42.589804 3262 ts_tablet_manager.cc:1918] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:49:42.589993 3262 log.cc:1196] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/027e4d0a55b64f4fae45ed53cb311e2f
I20260924 00:49:42.590021 3260 ts_tablet_manager.cc:1939] T 027e4d0a55b64f4fae45ed53cb311e2f P 534411c11bc44528aad2fe49b731f91c: Deleting consensus metadata
I20260924 00:49:42.590622 3262 ts_tablet_manager.cc:1939] T 027e4d0a55b64f4fae45ed53cb311e2f P dbb993b695b2428890c44c57805ffd8e: Deleting consensus metadata
I20260924 00:49:42.591079 2647 catalog_manager.cc:4841] TS 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785): tablet 027e4d0a55b64f4fae45ed53cb311e2f (table test-table [id=494f2aa33f584f51b1b33c97e2278572]) successfully deleted
I20260924 00:49:42.591267 2645 catalog_manager.cc:4841] TS 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365): tablet 027e4d0a55b64f4fae45ed53cb311e2f (table test-table [id=494f2aa33f584f51b1b33c97e2278572]) successfully deleted
I20260924 00:49:42.591753 2648 catalog_manager.cc:4841] TS dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787): tablet 027e4d0a55b64f4fae45ed53cb311e2f (table test-table [id=494f2aa33f584f51b1b33c97e2278572]) successfully deleted
Sep 24 00:49:42 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:42.613426 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:42.595235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.0.222.62:45591)
0924 00:49:42.595841 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:42.595848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:42.596002 (+ 154us) server_negotiation.cc:407] Connection header received
0924 00:49:42.596156 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:42.596160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:42.596221 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:42.596310 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:42.597665 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:42.598715 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:42.600186 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:42.600730 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:42.600842 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:42.606377 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:42.606384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:42.606387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:42.606430 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:42.610305 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:42.611140 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:42.611145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:42.611147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:42.611340 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:42.611757 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:42.611761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:42.611764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:42.612094 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:42.612383 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:42.612656 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 00:49:42.612916 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":323,"threads_started":1}
I20260924 00:49:42.617637 2659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:49:42.618438 2659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:49:42.626102 2800 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba218df14aa4253996ed0a6c85d72b3 (DEFAULT_TABLE table=test-table [id=37e2629126b04f48b930f4bc81604300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:42.626443 3083 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba218df14aa4253996ed0a6c85d72b3 (DEFAULT_TABLE table=test-table [id=37e2629126b04f48b930f4bc81604300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:42.626785 2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba218df14aa4253996ed0a6c85d72b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:42.626689 2942 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba218df14aa4253996ed0a6c85d72b3 (DEFAULT_TABLE table=test-table [id=37e2629126b04f48b930f4bc81604300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:42.626907 3083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba218df14aa4253996ed0a6c85d72b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:42.627194 2942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba218df14aa4253996ed0a6c85d72b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:42.639431 3272 tablet_bootstrap.cc:492] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: Bootstrap starting.
I20260924 00:49:42.640266 3271 tablet_bootstrap.cc:492] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: Bootstrap starting.
I20260924 00:49:42.640985 3273 tablet_bootstrap.cc:492] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: Bootstrap starting.
I20260924 00:49:42.642902 3272 tablet_bootstrap.cc:654] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:42.643370 3271 tablet_bootstrap.cc:654] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:42.643384 3273 tablet_bootstrap.cc:654] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:42.647153 3271 tablet_bootstrap.cc:492] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: No bootstrap required, opened a new log
I20260924 00:49:42.647397 3271 ts_tablet_manager.cc:1397] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260924 00:49:42.648777 3271 raft_consensus.cc:348] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.649125 3271 raft_consensus.cc:374] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:42.649315 3271 raft_consensus.cc:729] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb993b695b2428890c44c57805ffd8e, State: Initialized, Role: FOLLOWER
I20260924 00:49:42.649642 3271 consensus_queue.cc:260] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.651268 3271 ts_tablet_manager.cc:1428] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 00:49:42.651834 3273 tablet_bootstrap.cc:492] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: No bootstrap required, opened a new log
I20260924 00:49:42.652069 3273 ts_tablet_manager.cc:1397] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260924 00:49:42.652314 3272 tablet_bootstrap.cc:492] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: No bootstrap required, opened a new log
I20260924 00:49:42.652559 3272 ts_tablet_manager.cc:1397] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260924 00:49:42.653174 3273 raft_consensus.cc:348] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.653497 3273 raft_consensus.cc:374] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:42.653682 3273 raft_consensus.cc:729] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534411c11bc44528aad2fe49b731f91c, State: Initialized, Role: FOLLOWER
I20260924 00:49:42.653781 3272 raft_consensus.cc:348] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.654112 3272 raft_consensus.cc:374] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:42.653988 3273 consensus_queue.cc:260] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.654285 3272 raft_consensus.cc:729] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac3deb94f5d48a1b5106e76eef69daa, State: Initialized, Role: FOLLOWER
I20260924 00:49:42.654596 3272 consensus_queue.cc:260] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.655341 3272 ts_tablet_manager.cc:1428] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:49:42.655833 3273 ts_tablet_manager.cc:1428] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260924 00:49:42.744236 3237 raft_consensus.cc:482] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:42.744578 3237 raft_consensus.cc:504] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.745522 3237 leader_election.cc:288] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365), dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787)
I20260924 00:49:42.746070 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba218df14aa4253996ed0a6c85d72b3" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c" is_pre_election: true
I20260924 00:49:42.746145 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba218df14aa4253996ed0a6c85d72b3" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb993b695b2428890c44c57805ffd8e" is_pre_election: true
I20260924 00:49:42.746398 2961 raft_consensus.cc:2454] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 0.
I20260924 00:49:42.746484 3108 raft_consensus.cc:2454] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 0.
I20260924 00:49:42.747046 2749 leader_election.cc:302] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ac3deb94f5d48a1b5106e76eef69daa, 534411c11bc44528aad2fe49b731f91c, dbb993b695b2428890c44c57805ffd8e; no voters:
I20260924 00:49:42.747437 3237 raft_consensus.cc:2781] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:42.747617 3237 raft_consensus.cc:482] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:42.747750 3237 raft_consensus.cc:3037] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:42.794756 3237 raft_consensus.cc:504] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.795615 3237 leader_election.cc:288] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 election: Requested vote from peers 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365), dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787)
I20260924 00:49:42.796280 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba218df14aa4253996ed0a6c85d72b3" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c"
I20260924 00:49:42.796319 3108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba218df14aa4253996ed0a6c85d72b3" candidate_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb993b695b2428890c44c57805ffd8e"
I20260924 00:49:42.796593 2961 raft_consensus.cc:3037] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:42.796629 3108 raft_consensus.cc:3037] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:42.895413 3108 raft_consensus.cc:2454] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 1.
I20260924 00:49:42.895467 2961 raft_consensus.cc:2454] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac3deb94f5d48a1b5106e76eef69daa in term 1.
I20260924 00:49:42.896036 2749 leader_election.cc:302] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac3deb94f5d48a1b5106e76eef69daa, 534411c11bc44528aad2fe49b731f91c; no voters:
I20260924 00:49:42.896495 3237 raft_consensus.cc:2781] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:42.896687 3237 raft_consensus.cc:686] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 LEADER]: Becoming Leader. State: Replica: 3ac3deb94f5d48a1b5106e76eef69daa, State: Running, Role: LEADER
I20260924 00:49:42.897069 3237 consensus_queue.cc:237] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:42.900774 2659 catalog_manager.cc:5495] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1). New cstate: current_term: 1 leader_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } health_report { overall_health: UNKNOWN } } }
I20260924 00:49:42.953715 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:42.948302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.0.222.1:41785)
0924 00:49:42.949034 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:42.949038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:42.949057 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:49:42.949112 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:42.949116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:42.949176 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:42.949292 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:42.949738 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:42.950545 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:42.951905 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:42.952327 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:42.952438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:42.952927 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:42.953039 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:42.953116 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:42.953302 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 00:49:42.953388 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":344,"threads_started":1}
I20260924 00:49:42.963047 3108 raft_consensus.cc:1261] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Refusing update from remote peer 3ac3deb94f5d48a1b5106e76eef69daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:42.963007 2961 raft_consensus.cc:1261] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Refusing update from remote peer 3ac3deb94f5d48a1b5106e76eef69daa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:42.963598 3237 consensus_queue.cc:1035] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:42.963912 3282 consensus_queue.cc:1035] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [LEADER]: Connected to new peer: Peer: permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:43.017884 2661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:43.018174 2661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:43.021760 2661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: Sending DeleteTablet for 3 replicas of tablet 5ba218df14aa4253996ed0a6c85d72b3
I20260924 00:49:43.022763 2800 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ba218df14aa4253996ed0a6c85d72b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:43 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:45608
I20260924 00:49:43.023016 3083 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ba218df14aa4253996ed0a6c85d72b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:43 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40778
I20260924 00:49:43.023027 2942 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ba218df14aa4253996ed0a6c85d72b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:43 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:39428
I20260924 00:49:43.023140 3261 tablet_replica.cc:330] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: stopping tablet replica
I20260924 00:49:43.023342 3262 tablet_replica.cc:330] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: stopping tablet replica
I20260924 00:49:43.023432 3260 tablet_replica.cc:330] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: stopping tablet replica
I20260924 00:49:43.023420 3261 raft_consensus.cc:2229] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:49:43.023592 3262 raft_consensus.cc:2229] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:43.023710 3260 raft_consensus.cc:2229] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:43.023800 3261 raft_consensus.cc:2258] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:43.023834 3262 raft_consensus.cc:2258] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:43.023984 3260 raft_consensus.cc:2258] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:43.024286 2661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 00:49:43.025005 3261 ts_tablet_manager.cc:1905] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:43.025005 3262 ts_tablet_manager.cc:1905] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:49:43.025097 2661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:49:43.025573 3260 ts_tablet_manager.cc:1905] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:43.032823 2800 tablet_service.cc:1467] Processing CreateTablet for tablet a4d5c7183630423da82e8038bb1430e2 (DEFAULT_TABLE table=test-table [id=85adf3c2ef404dcdb102545beb547a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:43.033639 3083 tablet_service.cc:1467] Processing CreateTablet for tablet a4d5c7183630423da82e8038bb1430e2 (DEFAULT_TABLE table=test-table [id=85adf3c2ef404dcdb102545beb547a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:43.033746 2942 tablet_service.cc:1467] Processing CreateTablet for tablet a4d5c7183630423da82e8038bb1430e2 (DEFAULT_TABLE table=test-table [id=85adf3c2ef404dcdb102545beb547a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:49:43.101627 3083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d5c7183630423da82e8038bb1430e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:43.102151 2942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d5c7183630423da82e8038bb1430e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:43.106554 2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d5c7183630423da82e8038bb1430e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:43.110000 3271 tablet_bootstrap.cc:492] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: Bootstrap starting.
I20260924 00:49:43.111634 3273 tablet_bootstrap.cc:492] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: Bootstrap starting.
I20260924 00:49:43.112545 3271 tablet_bootstrap.cc:654] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:43.113826 3273 tablet_bootstrap.cc:654] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:43.115660 3271 tablet_bootstrap.cc:492] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: No bootstrap required, opened a new log
I20260924 00:49:43.115967 3271 ts_tablet_manager.cc:1397] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 00:49:43.116604 3273 tablet_bootstrap.cc:492] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: No bootstrap required, opened a new log
I20260924 00:49:43.116812 3273 ts_tablet_manager.cc:1397] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260924 00:49:43.116943 3271 raft_consensus.cc:348] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.117282 3271 raft_consensus.cc:374] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:43.117401 3271 raft_consensus.cc:729] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb993b695b2428890c44c57805ffd8e, State: Initialized, Role: FOLLOWER
I20260924 00:49:43.117712 3271 consensus_queue.cc:260] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.117723 3273 raft_consensus.cc:348] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.118028 3273 raft_consensus.cc:374] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:43.118013 3252 raft_consensus.cc:482] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:49:43.118206 3273 raft_consensus.cc:729] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534411c11bc44528aad2fe49b731f91c, State: Initialized, Role: FOLLOWER
I20260924 00:49:43.118427 3252 raft_consensus.cc:504] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.118597 3273 consensus_queue.cc:260] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.119331 3252 leader_election.cc:288] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785), 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365)
I20260924 00:49:43.119375 3273 ts_tablet_manager.cc:1428] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:49:43.119792 3271 ts_tablet_manager.cc:1428] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260924 00:49:43.119992 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d5c7183630423da82e8038bb1430e2" candidate_uuid: "dbb993b695b2428890c44c57805ffd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c" is_pre_election: true
I20260924 00:49:43.120245 2961 raft_consensus.cc:2454] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbb993b695b2428890c44c57805ffd8e in term 0.
W20260924 00:49:43.120657 3035 leader_election.cc:341] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785): Not found: Tablet not found: a4d5c7183630423da82e8038bb1430e2
I20260924 00:49:43.121042 3033 leader_election.cc:302] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 534411c11bc44528aad2fe49b731f91c, dbb993b695b2428890c44c57805ffd8e; no voters: 3ac3deb94f5d48a1b5106e76eef69daa
I20260924 00:49:43.121397 3252 raft_consensus.cc:2781] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:49:43.121577 3252 raft_consensus.cc:482] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:49:43.121722 3252 raft_consensus.cc:3037] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:43.206382 3262 ts_tablet_manager.cc:1918] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:43.206707 3262 log.cc:1196] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/5ba218df14aa4253996ed0a6c85d72b3
I20260924 00:49:43.206698 3260 ts_tablet_manager.cc:1918] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:43.206951 3260 log.cc:1196] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/5ba218df14aa4253996ed0a6c85d72b3
I20260924 00:49:43.207417 3262 ts_tablet_manager.cc:1939] T 5ba218df14aa4253996ed0a6c85d72b3 P dbb993b695b2428890c44c57805ffd8e: Deleting consensus metadata
I20260924 00:49:43.207574 3260 ts_tablet_manager.cc:1939] T 5ba218df14aa4253996ed0a6c85d72b3 P 534411c11bc44528aad2fe49b731f91c: Deleting consensus metadata
I20260924 00:49:43.207819 3252 raft_consensus.cc:504] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.209569 2961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d5c7183630423da82e8038bb1430e2" candidate_uuid: "dbb993b695b2428890c44c57805ffd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534411c11bc44528aad2fe49b731f91c"
I20260924 00:49:43.209873 2961 raft_consensus.cc:3037] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:43.212258 3261 ts_tablet_manager.cc:1918] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:49:43.212358 3252 leader_election.cc:288] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 election: Requested vote from peers 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785), 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365)
I20260924 00:49:43.212575 3261 log.cc:1196] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/5ba218df14aa4253996ed0a6c85d72b3
I20260924 00:49:43.213171 2820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d5c7183630423da82e8038bb1430e2" candidate_uuid: "dbb993b695b2428890c44c57805ffd8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac3deb94f5d48a1b5106e76eef69daa"
I20260924 00:49:43.213954 3261 ts_tablet_manager.cc:1939] T 5ba218df14aa4253996ed0a6c85d72b3 P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting consensus metadata
W20260924 00:49:43.214046 3035 leader_election.cc:341] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785): Illegal state: must be running to vote when last-logged opid is not known
I20260924 00:49:43.214407 2648 catalog_manager.cc:4841] TS dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787): tablet 5ba218df14aa4253996ed0a6c85d72b3 (table test-table [id=37e2629126b04f48b930f4bc81604300]) successfully deleted
I20260924 00:49:43.214690 3289 tablet_bootstrap.cc:492] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa: Bootstrap starting.
I20260924 00:49:43.217360 3289 tablet_bootstrap.cc:654] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:43.220517 3289 tablet_bootstrap.cc:492] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa: No bootstrap required, opened a new log
I20260924 00:49:43.220729 3289 ts_tablet_manager.cc:1397] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260924 00:49:43.221896 3289 raft_consensus.cc:348] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.222277 3289 raft_consensus.cc:374] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:43.222445 3289 raft_consensus.cc:729] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac3deb94f5d48a1b5106e76eef69daa, State: Initialized, Role: FOLLOWER
I20260924 00:49:43.222723 3289 consensus_queue.cc:260] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.223526 3289 ts_tablet_manager.cc:1428] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260924 00:49:43.224197 2645 catalog_manager.cc:4841] TS 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365): tablet 5ba218df14aa4253996ed0a6c85d72b3 (table test-table [id=37e2629126b04f48b930f4bc81604300]) successfully deleted
I20260924 00:49:43.258003 2647 catalog_manager.cc:4841] TS 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785): tablet 5ba218df14aa4253996ed0a6c85d72b3 (table test-table [id=37e2629126b04f48b930f4bc81604300]) successfully deleted
I20260924 00:49:43.306794 2961 raft_consensus.cc:2454] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbb993b695b2428890c44c57805ffd8e in term 1.
I20260924 00:49:43.307557 3033 leader_election.cc:302] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 534411c11bc44528aad2fe49b731f91c, dbb993b695b2428890c44c57805ffd8e; no voters: 3ac3deb94f5d48a1b5106e76eef69daa
I20260924 00:49:43.308007 3252 raft_consensus.cc:2781] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:43.308220 3252 raft_consensus.cc:686] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 1 LEADER]: Becoming Leader. State: Replica: dbb993b695b2428890c44c57805ffd8e, State: Running, Role: LEADER
I20260924 00:49:43.308614 3252 consensus_queue.cc:237] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } }
I20260924 00:49:43.311838 2661 catalog_manager.cc:5495] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e reported cstate change: term changed from 0 to 1, leader changed from <none> to dbb993b695b2428890c44c57805ffd8e (127.0.222.3). New cstate: current_term: 1 leader_uuid: "dbb993b695b2428890c44c57805ffd8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb993b695b2428890c44c57805ffd8e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 35787 } health_report { overall_health: HEALTHY } } }
I20260924 00:49:43.364919 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:43.357664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.0.222.3:35787)
0924 00:49:43.358596 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:43.358601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:43.358632 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:49:43.358692 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:43.358697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:43.358763 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:43.358868 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:49:43.359503 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:43.360650 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:43.362476 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:43.363113 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:43.363231 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:43.363875 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:49:43.364011 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:49:43.364090 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:43.364261 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 00:49:43.364342 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":580,"threads_started":1}
I20260924 00:49:43.375090 2961 raft_consensus.cc:1261] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Refusing update from remote peer dbb993b695b2428890c44c57805ffd8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:43.375242 2820 raft_consensus.cc:3037] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:43.375707 3288 consensus_queue.cc:1035] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "534411c11bc44528aad2fe49b731f91c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:49:43.396809 3241 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 00:49:43.410774 2820 raft_consensus.cc:1261] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Refusing update from remote peer dbb993b695b2428890c44c57805ffd8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:49:43.411618 3252 consensus_queue.cc:1035] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac3deb94f5d48a1b5106e76eef69daa" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:49:45.147784 2659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:45.148519 2659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:49:45.180986 2659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de5a7bd8e5244a57bf77787775ad2143: Sending DeleteTablet for 3 replicas of tablet a4d5c7183630423da82e8038bb1430e2
I20260924 00:49:45.187882 2800 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d5c7183630423da82e8038bb1430e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:45 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:45608
I20260924 00:49:45.188237 2942 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d5c7183630423da82e8038bb1430e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:45 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:39428
I20260924 00:49:45.191133 3083 tablet_service.cc:1514] Processing DeleteTablet for tablet a4d5c7183630423da82e8038bb1430e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:49:45 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40778
I20260924 00:49:45.191705 3321 tablet_replica.cc:330] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa: stopping tablet replica
I20260924 00:49:45.192665 3321 raft_consensus.cc:2229] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:45.206918 3322 tablet_replica.cc:330] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: stopping tablet replica
I20260924 00:49:45.207281 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2733
I20260924 00:49:45.211369 3323 tablet_replica.cc:330] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: stopping tablet replica
I20260924 00:49:45.204780 3321 raft_consensus.cc:2258] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:45.220367 3322 raft_consensus.cc:2229] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:49:45.227604 3323 raft_consensus.cc:2229] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:49:45.229128 3321 ts_tablet_manager.cc:1905] T a4d5c7183630423da82e8038bb1430e2 P 3ac3deb94f5d48a1b5106e76eef69daa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:45.233263 3323 raft_consensus.cc:2258] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:45.222229 3322 raft_consensus.cc:2258] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:49:45.248816 3323 ts_tablet_manager.cc:1905] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:49:45.255666 3322 ts_tablet_manager.cc:1905] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 00:49:45 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:49:45.366453 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:45.218179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.0.222.1:41785)
0924 00:49:45.221627 (+ 3448us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:45.221713 (+ 86us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:45.227496 (+ 5783us) server_negotiation.cc:407] Connection header received
0924 00:49:45.228489 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:45.228537 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:45.229123 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:45.229829 (+ 706us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:45.247303 (+ 17474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:45.255230 (+ 7927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:45.268940 (+ 13710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:45.274613 (+ 5673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:45.275674 (+ 1061us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:45.310445 (+ 34771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:45.310501 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:45.310557 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:45.310805 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:45.328184 (+ 17379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:45.349480 (+ 21296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:45.349520 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:45.349534 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:45.350957 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:45.354481 (+ 3524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:45.354513 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:45.354517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:45.357583 (+ 3066us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:45.361640 (+ 4057us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:45.362753 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0924 00:49:45.363896 (+ 1143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2414,"thread_start_us":5341,"threads_started":1}
W20260924 00:49:45.371516 2866 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45638
W20260924 00:49:45.377355 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2733: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:45.379042 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 2733
I20260924 00:49:45.447584 3323 ts_tablet_manager.cc:1918] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:49:45.448858 3323 log.cc:1196] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/a4d5c7183630423da82e8038bb1430e2
I20260924 00:49:45.451277 3322 ts_tablet_manager.cc:1918] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:49:45.451485 3323 ts_tablet_manager.cc:1939] T a4d5c7183630423da82e8038bb1430e2 P dbb993b695b2428890c44c57805ffd8e: Deleting consensus metadata
I20260924 00:49:45.452476 3322 log.cc:1196] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/a4d5c7183630423da82e8038bb1430e2
I20260924 00:49:45.455971 3322 ts_tablet_manager.cc:1939] T a4d5c7183630423da82e8038bb1430e2 P 534411c11bc44528aad2fe49b731f91c: Deleting consensus metadata
I20260924 00:49:45.460335 2648 catalog_manager.cc:4841] TS dbb993b695b2428890c44c57805ffd8e (127.0.222.3:35787): tablet a4d5c7183630423da82e8038bb1430e2 (table test-table [id=85adf3c2ef404dcdb102545beb547a66]) successfully deleted
I20260924 00:49:45.464980 2645 catalog_manager.cc:4841] TS 534411c11bc44528aad2fe49b731f91c (127.0.222.2:39365): tablet a4d5c7183630423da82e8038bb1430e2 (table test-table [id=85adf3c2ef404dcdb102545beb547a66]) successfully deleted
I20260924 00:49:45.518082 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2875
W20260924 00:49:45.518574 2647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:41785 (error 108)
I20260924 00:49:45.522013 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:45.520760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.0.1:45648)
0924 00:49:45.521485 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:45.521687 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":411,"threads_started":1}
W20260924 00:49:45.522261 2647 catalog_manager.cc:4584] TS 3ac3deb94f5d48a1b5106e76eef69daa (127.0.222.1:41785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4d5c7183630423da82e8038bb1430e2: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41785: connect: Connection refused (error 111)
Sep 24 00:49:45 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:49:45.537774 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:45.519276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.222.2:39365)
0924 00:49:45.520112 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:45.520118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:45.520149 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:49:45.520262 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:45.520271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:45.520382 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:45.520531 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:45.522633 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:45.523553 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:45.525057 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:45.525553 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:45.525650 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:45.530294 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:45.530310 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:45.530332 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:45.530388 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:45.534285 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:45.535092 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:45.535102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:45.535104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:45.535269 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:45.535760 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:45.535765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:45.535767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:45.536299 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:45.536720 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:45.537050 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 00:49:45.537398 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":393,"threads_started":1}
W20260924 00:49:45.538684 3008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39434
W20260924 00:49:45.539481 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2875: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:45.539669 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 2875
I20260924 00:49:45.554286 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3017
Sep 24 00:49:45 dist-test-slave-rtvz krb5kdc[2616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:49:45.572368 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:45.555311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.222.3:35787)
0924 00:49:45.555945 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:45.555950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:45.555975 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:49:45.556062 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:45.556066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:45.556166 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:45.556318 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:45.557855 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:45.558755 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:45.560200 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:45.560684 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:45.560784 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:45.565317 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:45.565334 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:45.565351 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:45.565408 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:45.568836 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:45.569668 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:45.569675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:45.569678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:45.569868 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:45.570235 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:45.570240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:45.570244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:45.570699 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:45.571055 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:45.571743 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 00:49:45.572015 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":342,"threads_started":1}
W20260924 00:49:45.573293 3149 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40810
W20260924 00:49:45.574079 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3017: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:45.574235 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 3017
I20260924 00:49:45.577011 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:45.576586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.0.1:45652)
0924 00:49:45.576766 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:45.576849 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 00:49:45.578101 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:45.577710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.0.1:45666)
0924 00:49:45.577863 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:45.577953 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 00:49:45.594565 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 2629
I20260924 00:49:45.596051 2702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:45.767722 3328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:45.766955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:41785 (local address 127.0.0.1:45674)
0924 00:49:45.767166 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:45.767526 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 00:49:45.767853 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 2629
2026-09-24T00:49:45Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9380 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:46 dist-test-slave-rtvz krb5kdc[3333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:46 dist-test-slave-rtvz krb5kdc[3333](info): set up 2 sockets
Sep 24 00:49:46 dist-test-slave-rtvz krb5kdc[3333](info): commencing operation
krb5kdc: starting...
W20260924 00:49:47.869325 888 mini_kdc.cc:121] Time spent starting KDC: real 2.056s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:49:47 dist-test-slave-rtvz krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:49:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:49:48.056514 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46567
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:46567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:48.392477 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:48.392774 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:48.392901 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:48.400381 3346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:49:48.400501 3346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:48.400555 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:48.400635 3346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:49:48.400688 3346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:49:48.410935 3346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:46567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:48.412356 3346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:48.413925 3346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:49:48.426160 3346 server_base.cc:1034] running on GCE node
W20260924 00:49:48.426149 3351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:48.426234 3352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:48.426198 3354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:48.427452 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:48.428962 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:48.430182 3346 hybrid_clock.cc:648] HybridClock initialized: now 1790210988430107 us; error 59 us; skew 500 ppm
I20260924 00:49:48.430531 3346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:48 dist-test-slave-rtvz krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:48.433894 3346 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:49:48.435456 3346 webserver.cc:460] Webserver started at http://127.0.222.62:42387/ using document root <none> and password file <none>
I20260924 00:49:48.436185 3346 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:48.436318 3346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:48.436648 3346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:48.439713 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "9151196db2974dce96c9908c61b8d72b"
format_stamp: "Formatted at 2026-09-24 00:49:48 on dist-test-slave-rtvz"
server_key: "e88344e1a5f8c5aa5d789a630490006a"
server_key_iv: "2dbad1a80ff2b3d6a2318ecbf3bae69f"
server_key_version: "encryptionkey@0"
I20260924 00:49:48.440478 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "9151196db2974dce96c9908c61b8d72b"
format_stamp: "Formatted at 2026-09-24 00:49:48 on dist-test-slave-rtvz"
server_key: "e88344e1a5f8c5aa5d789a630490006a"
server_key_iv: "2dbad1a80ff2b3d6a2318ecbf3bae69f"
server_key_version: "encryptionkey@0"
I20260924 00:49:48.445863 3346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 00:49:48.449263 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:48.450569 3346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:48.450745 3346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "9151196db2974dce96c9908c61b8d72b"
format_stamp: "Formatted at 2026-09-24 00:49:48 on dist-test-slave-rtvz"
server_key: "e88344e1a5f8c5aa5d789a630490006a"
server_key_iv: "2dbad1a80ff2b3d6a2318ecbf3bae69f"
server_key_version: "encryptionkey@0"
I20260924 00:49:48.450932 3346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:48.473713 3346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:48.478263 3346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:48.478554 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:48.497848 3346 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:46567
I20260924 00:49:48.497877 3413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:46567 every 8 connection(s)
I20260924 00:49:48.499336 3346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:49:48.503425 3414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:48.505250 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 3346
I20260924 00:49:48.505475 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:49:48.506367 888 external_mini_cluster.cc:1428] Setting key c2a96ecb8fd2ef807752b0492eba2a40
I20260924 00:49:48.516714 3414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b: Bootstrap starting.
I20260924 00:49:48.520601 3414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:48.521876 3414 log.cc:824] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b: Log is configured to *not* fsync() on all Append() calls
Sep 24 00:49:48 dist-test-slave-rtvz krb5kdc[3333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:48.524665 3414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b: No bootstrap required, opened a new log
I20260924 00:49:48.530102 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:48.508638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.0.222.62:46567)
0924 00:49:48.509602 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:48.509616 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:48.509670 (+ 54us) server_negotiation.cc:407] Connection header received
0924 00:49:48.511445 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:48.511470 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:48.511892 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:48.512246 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:48.514346 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:48.515797 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:48.517546 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:48.518261 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:48.518418 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:48.523068 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:48.523076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:48.523092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:48.523127 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:48.526094 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:48.526779 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:48.526785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:48.526791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:48.526960 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:48.527312 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:48.527315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:48.527317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:48.527872 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:48.528167 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:48.528608 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 00:49:48.529026 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":466,"threads_started":1}
I20260924 00:49:48.532804 3414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9151196db2974dce96c9908c61b8d72b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46567 } }
I20260924 00:49:48.533254 3414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:48.533350 3414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9151196db2974dce96c9908c61b8d72b, State: Initialized, Role: FOLLOWER
I20260924 00:49:48.533874 3414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9151196db2974dce96c9908c61b8d72b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46567 } }
I20260924 00:49:48.534078 3414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:49:48.534174 3414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:49:48.534292 3414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:48.536422 3414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9151196db2974dce96c9908c61b8d72b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46567 } }
I20260924 00:49:48.536885 3414 leader_election.cc:302] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9151196db2974dce96c9908c61b8d72b; no voters:
I20260924 00:49:48.537528 3414 leader_election.cc:288] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:49:48.537686 3419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:48.538581 3419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [term 1 LEADER]: Becoming Leader. State: Replica: 9151196db2974dce96c9908c61b8d72b, State: Running, Role: LEADER
I20260924 00:49:48.539268 3419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9151196db2974dce96c9908c61b8d72b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46567 } }
I20260924 00:49:48.539800 3414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:49:48.542393 3421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9151196db2974dce96c9908c61b8d72b. Latest consensus state: current_term: 1 leader_uuid: "9151196db2974dce96c9908c61b8d72b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9151196db2974dce96c9908c61b8d72b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46567 } } }
I20260924 00:49:48.542414 3420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9151196db2974dce96c9908c61b8d72b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9151196db2974dce96c9908c61b8d72b" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46567 } } }
I20260924 00:49:48.543010 3421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:48.543033 3420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:48.546437 3425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:49:48.553009 3425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:49:48.563302 3425 catalog_manager.cc:1347] Generated new cluster ID: 306c5b8ce26a49f6a5413b051d961653
I20260924 00:49:48.563454 3425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:49:48.577126 3425 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:49:48.579206 3425 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:49:48.590659 3425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9151196db2974dce96c9908c61b8d72b: Generated new TSK 0
I20260924 00:49:48.591655 3425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:49:48.649310 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46567
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:49.007639 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:49.007948 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:49.008069 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:49.016004 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:49.016136 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:49.016283 3440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:49:49.026466 3440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:49.027798 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:49.029543 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:49:49.046461 3440 server_base.cc:1034] running on GCE node
W20260924 00:49:49.046427 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:49.046383 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:49.046741 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:49.047612 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:49.048663 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:49.049927 3440 hybrid_clock.cc:648] HybridClock initialized: now 1790210989049887 us; error 83 us; skew 500 ppm
I20260924 00:49:49.050298 3440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:49 dist-test-slave-rtvz krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:49.054046 3440 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:49:49.055823 3440 webserver.cc:460] Webserver started at http://127.0.222.1:37407/ using document root <none> and password file <none>
I20260924 00:49:49.056572 3440 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:49.056704 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:49.057032 3440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:49.060261 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "9fc10ed422d64a369e33bfe863356135"
format_stamp: "Formatted at 2026-09-24 00:49:49 on dist-test-slave-rtvz"
server_key: "dc81e1492b8dced88f09459af9eae832"
server_key_iv: "76f5f96448551a8667eb6dbba46dbfeb"
server_key_version: "encryptionkey@0"
I20260924 00:49:49.061151 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "9fc10ed422d64a369e33bfe863356135"
format_stamp: "Formatted at 2026-09-24 00:49:49 on dist-test-slave-rtvz"
server_key: "dc81e1492b8dced88f09459af9eae832"
server_key_iv: "76f5f96448551a8667eb6dbba46dbfeb"
server_key_version: "encryptionkey@0"
I20260924 00:49:49.066948 3440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 00:49:49.070842 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:49.072355 3440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:49:49.072542 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "9fc10ed422d64a369e33bfe863356135"
format_stamp: "Formatted at 2026-09-24 00:49:49 on dist-test-slave-rtvz"
server_key: "dc81e1492b8dced88f09459af9eae832"
server_key_iv: "76f5f96448551a8667eb6dbba46dbfeb"
server_key_version: "encryptionkey@0"
I20260924 00:49:49.072743 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:49.090498 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:49.094730 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:49.095016 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:49.096069 3440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:49.097555 3440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:49.097637 3440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:49.097764 3440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:49.097816 3440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:49.133738 3440 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44133
I20260924 00:49:49.133805 3568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44133 every 8 connection(s)
I20260924 00:49:49.135116 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:49:49.140192 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 3440
I20260924 00:49:49.140390 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:49:49.141063 888 external_mini_cluster.cc:1428] Setting key f6abcb6301a7e4f2a5236fb0d3c0c218
Sep 24 00:49:49 dist-test-slave-rtvz krb5kdc[3333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:49.156296 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:49.137716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45587 (local address 127.0.222.62:46567)
0924 00:49:49.138137 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:49.138142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:49.139129 (+ 987us) server_negotiation.cc:407] Connection header received
0924 00:49:49.141107 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:49.141112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:49.141225 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:49.141336 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:49.142997 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:49.144072 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:49.145647 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:49.146211 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:49.146336 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:49.150228 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:49.150234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:49.150237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:49.150275 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:49.153013 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:49.153922 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:49.153927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:49.153931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:49.154063 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:49.154505 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:49.154509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:49.154511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:49.154810 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:49.154940 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:49.155689 (+ 749us) server_negotiation.cc:299] Negotiation successful
0924 00:49:49.155931 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":216,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:49:49.157356 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:49.138164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46567 (local address 127.0.222.1:45587)
0924 00:49:49.138968 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:49.139008 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:49.140789 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:49.141589 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:49.141598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:49.142222 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:49.142823 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:49.142850 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:49.144224 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:49.144227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:49.145519 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:49.145532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:49.146417 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:49.146422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:49.146650 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:49.147341 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:49.147364 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:49.149940 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:49.153187 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:49.153225 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:49.153246 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:49.153795 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:49.154172 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:49.154177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:49.154181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:49.154388 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:49.154939 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:49.154948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:49.155302 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 00:49:49.155546 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 00:49:49.156149 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":300,"threads_started":1}
I20260924 00:49:49.158989 3569 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46567
I20260924 00:49:49.159303 3569 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260924 00:49:49.160099 3569 heartbeater.cc:511] Master 127.0.222.62:46567 requested a full tablet report, sending...
I20260924 00:49:49.162241 3378 ts_manager.cc:194] Registered new tserver with Master: 9fc10ed422d64a369e33bfe863356135 (127.0.222.1:44133)
I20260924 00:49:49.163856 3378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:45587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:49:49.181347 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46567
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:49.536294 3575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:49.536612 3575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:49.536748 3575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:49.544464 3575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:49.544593 3575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:49.544741 3575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:49:49.555078 3575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:49.556488 3575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:49.557948 3575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:49.571879 3580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:49.571856 3581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:49.571785 3583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:49.574551 3575 server_base.cc:1034] running on GCE node
I20260924 00:49:49.575179 3575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:49.576160 3575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:49.577384 3575 hybrid_clock.cc:648] HybridClock initialized: now 1790210989577295 us; error 59 us; skew 500 ppm
I20260924 00:49:49.577761 3575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:49 dist-test-slave-rtvz krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:49.581038 3575 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:49:49.582633 3575 webserver.cc:460] Webserver started at http://127.0.222.2:38955/ using document root <none> and password file <none>
I20260924 00:49:49.583377 3575 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:49.583518 3575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:49.583837 3575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:49.586810 3575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "729984b1329f4ad29d8adf9eb32723ee"
format_stamp: "Formatted at 2026-09-24 00:49:49 on dist-test-slave-rtvz"
server_key: "cc2cee0ca5b37bf341291cf5de738507"
server_key_iv: "e0380d4aaa504805f852f73868d7c86a"
server_key_version: "encryptionkey@0"
I20260924 00:49:49.587625 3575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "729984b1329f4ad29d8adf9eb32723ee"
format_stamp: "Formatted at 2026-09-24 00:49:49 on dist-test-slave-rtvz"
server_key: "cc2cee0ca5b37bf341291cf5de738507"
server_key_iv: "e0380d4aaa504805f852f73868d7c86a"
server_key_version: "encryptionkey@0"
I20260924 00:49:49.593312 3575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 00:49:49.597136 3590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:49.598536 3575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:49:49.598695 3575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "729984b1329f4ad29d8adf9eb32723ee"
format_stamp: "Formatted at 2026-09-24 00:49:49 on dist-test-slave-rtvz"
server_key: "cc2cee0ca5b37bf341291cf5de738507"
server_key_iv: "e0380d4aaa504805f852f73868d7c86a"
server_key_version: "encryptionkey@0"
I20260924 00:49:49.598910 3575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:49.616662 3575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:49.621699 3575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:49.622025 3575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:49.623059 3575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:49.624593 3575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:49.624681 3575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:49.624825 3575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:49.624893 3575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:49.662447 3575 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:40419
I20260924 00:49:49.662482 3702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:40419 every 8 connection(s)
I20260924 00:49:49.663862 3575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:49:49.669811 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 3575
I20260924 00:49:49.670007 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:49:49.670548 888 external_mini_cluster.cc:1428] Setting key e606c4268f9951d96b0336dff459af2d
Sep 24 00:49:49 dist-test-slave-rtvz krb5kdc[3333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:49.687359 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:49.666501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:36695 (local address 127.0.222.62:46567)
0924 00:49:49.666992 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:49.666997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:49.668019 (+ 1022us) server_negotiation.cc:407] Connection header received
0924 00:49:49.670074 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:49.670081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:49.670166 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:49.670260 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:49.672070 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:49.673125 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:49.675685 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:49.676182 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:49.676286 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:49.680425 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:49.680432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:49.680437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:49.680481 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:49.683905 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:49.684723 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:49.684728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:49.684731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:49.684913 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:49.685359 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:49.685363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:49.685367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:49.685717 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:49.685856 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:49.686663 (+ 807us) server_negotiation.cc:299] Negotiation successful
0924 00:49:49.686904 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":277,"threads_started":1}
I20260924 00:49:49.688189 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:49.667000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46567 (local address 127.0.222.2:36695)
0924 00:49:49.667798 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:49.667848 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:49.669710 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:49.670563 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:49.670575 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:49.671292 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:49.671898 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:49.671917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:49.673386 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:49.673392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:49.675498 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:49.675515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:49.676434 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:49.676441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:49.676743 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:49.677578 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:49.677613 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:49.680114 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:49.684079 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:49.684088 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:49.684104 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:49.684590 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:49.685014 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:49.685017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:49.685019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:49.685241 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:49.685839 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:49.685845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:49.686233 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 00:49:49.686511 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 00:49:49.686934 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":354,"threads_started":1}
I20260924 00:49:49.689764 3704 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46567
I20260924 00:49:49.690146 3704 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:49:49.691097 3704 heartbeater.cc:511] Master 127.0.222.62:46567 requested a full tablet report, sending...
I20260924 00:49:49.692713 3378 ts_manager.cc:194] Registered new tserver with Master: 729984b1329f4ad29d8adf9eb32723ee (127.0.222.2:40419)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:49:49.693789 3378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:36695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:49:49.716643 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46567
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:50.090876 3710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:50.091216 3710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:50.091347 3710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:50.099033 3710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:50.099187 3710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:50.099355 3710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:49:50.110255 3710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:50.111617 3710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:50.112972 3710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:50.125991 3718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:50.126036 3716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:50.126062 3715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:50.127210 3710 server_base.cc:1034] running on GCE node
I20260924 00:49:50.127892 3710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:50.128944 3710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:50.130191 3710 hybrid_clock.cc:648] HybridClock initialized: now 1790210990130103 us; error 69 us; skew 500 ppm
I20260924 00:49:50.130574 3710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:50 dist-test-slave-rtvz krb5kdc[3333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:50.134945 3710 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:49:50.136837 3710 webserver.cc:460] Webserver started at http://127.0.222.3:35751/ using document root <none> and password file <none>
I20260924 00:49:50.137794 3710 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:50.137957 3710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:50.138337 3710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:50.141808 3710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "5d7b55ea1d7d45ef880aec5e16f651ed"
format_stamp: "Formatted at 2026-09-24 00:49:50 on dist-test-slave-rtvz"
server_key: "c5523352021d8bb3293fa5e60f755dcb"
server_key_iv: "cd8e5e65fdf6af8d6fd048d8575ae867"
server_key_version: "encryptionkey@0"
I20260924 00:49:50.142575 3710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "5d7b55ea1d7d45ef880aec5e16f651ed"
format_stamp: "Formatted at 2026-09-24 00:49:50 on dist-test-slave-rtvz"
server_key: "c5523352021d8bb3293fa5e60f755dcb"
server_key_iv: "cd8e5e65fdf6af8d6fd048d8575ae867"
server_key_version: "encryptionkey@0"
I20260924 00:49:50.147913 3710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:49:50.151705 3725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:50.153126 3710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 00:49:50.153339 3710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "5d7b55ea1d7d45ef880aec5e16f651ed"
format_stamp: "Formatted at 2026-09-24 00:49:50 on dist-test-slave-rtvz"
server_key: "c5523352021d8bb3293fa5e60f755dcb"
server_key_iv: "cd8e5e65fdf6af8d6fd048d8575ae867"
server_key_version: "encryptionkey@0"
I20260924 00:49:50.153545 3710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:50.167047 3569 heartbeater.cc:503] Master 127.0.222.62:46567 was elected leader, sending a full tablet report...
I20260924 00:49:50.182394 3710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:50.187253 3710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:50.187522 3710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:50.188439 3710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:50.189971 3710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:50.190047 3710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:50.190133 3710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:50.190176 3710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:50.227420 3710 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:38301
I20260924 00:49:50.227494 3838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:38301 every 8 connection(s)
I20260924 00:49:50.228709 3710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:49:50.235623 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 3710
I20260924 00:49:50.235805 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:49:50.236387 888 external_mini_cluster.cc:1428] Setting key ef7819782837a19903158fcc255f77e1
Sep 24 00:49:50 dist-test-slave-rtvz krb5kdc[3333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:50.251364 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:50.231393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:44287 (local address 127.0.222.62:46567)
0924 00:49:50.231920 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:50.231927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:50.232858 (+ 931us) server_negotiation.cc:407] Connection header received
0924 00:49:50.234965 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:50.234970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:50.235037 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:50.235132 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:50.237346 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:50.238388 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:50.240381 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:50.240973 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:50.241078 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:50.244824 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:50.244829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:50.244833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:50.244870 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:50.247649 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:50.248517 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:50.248522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:50.248524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:50.248649 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:50.249055 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:50.249058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:50.249059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:50.249378 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:50.249506 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:50.250648 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0924 00:49:50.250908 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":240,"threads_started":1}
I20260924 00:49:50.252885 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:50.231994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46567 (local address 127.0.222.3:44287)
0924 00:49:50.232700 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:50.232738 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:50.234689 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:50.235404 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:50.235425 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:50.236270 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:50.237141 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:50.237164 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:50.238574 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:50.238578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:50.240203 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:50.240224 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:50.241226 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:50.241233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:50.241471 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:50.242140 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:50.242163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:50.244588 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:50.247786 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:50.247793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:50.247808 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:50.248377 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:50.248732 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:50.248735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:50.248737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:50.248973 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:50.249535 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:50.249542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:50.250115 (+ 573us) client_negotiation.cc:769] Sending connection context
0924 00:49:50.250479 (+ 364us) client_negotiation.cc:240] Negotiation successful
0924 00:49:50.251113 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":264,"threads_started":1}
I20260924 00:49:50.254631 3839 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46567
I20260924 00:49:50.255028 3839 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:50.256047 3839 heartbeater.cc:511] Master 127.0.222.62:46567 requested a full tablet report, sending...
I20260924 00:49:50.257790 3378 ts_manager.cc:194] Registered new tserver with Master: 5d7b55ea1d7d45ef880aec5e16f651ed (127.0.222.3:38301)
I20260924 00:49:50.258771 3378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:44287
I20260924 00:49:50.264962 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:49:50.294589 3841 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:49:50.291761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.0.222.62:46567)
0924 00:49:50.292042 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:50.292050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:50.292473 (+ 423us) server_negotiation.cc:407] Connection header received
0924 00:49:50.292623 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:50.292627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:50.292694 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:50.292778 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:50.294292 (+ 1514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42790: BlockingRecv error: recv got EOF from 127.0.0.1:42790 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 00:49:50.297817 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3440
W20260924 00:49:50.300841 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3440: Not authorized: Client connection negotiation failed: client connection to 127.0.222.1:44133: server requires authentication, but client does not have Kerberos credentials available
I20260924 00:49:50.301136 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 3440
I20260924 00:49:50.313482 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3575
W20260924 00:49:50.316203 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3575: Not authorized: Client connection negotiation failed: client connection to 127.0.222.2:40419: server requires authentication, but client does not have Kerberos credentials available
I20260924 00:49:50.316359 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 3575
I20260924 00:49:50.328553 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3710
W20260924 00:49:50.331655 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3710: Not authorized: Client connection negotiation failed: client connection to 127.0.222.3:38301: server requires authentication, but client does not have Kerberos credentials available
I20260924 00:49:50.331809 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 3710
I20260924 00:49:50.344089 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3346
I20260924 00:49:50.345491 3409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:50.452539 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 3346
2026-09-24T00:49:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4677 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:50 dist-test-slave-rtvz krb5kdc[3857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:50 dist-test-slave-rtvz krb5kdc[3857](info): set up 2 sockets
Sep 24 00:49:50 dist-test-slave-rtvz krb5kdc[3857](info): commencing operation
krb5kdc: starting...
W20260924 00:49:51.477164 888 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:49:51 dist-test-slave-rtvz krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:49:51.658409 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39803
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:41939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:39803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:51.976838 3871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:51.977172 3871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:51.977324 3871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:51.984946 3871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:49:51.985067 3871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:51.985121 3871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:51.985224 3871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:49:51.985291 3871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:49:51.995743 3871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:41939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:39803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:51.997067 3871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:51.998435 3871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:52.009991 3876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:52.010645 3879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:52.010675 3877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:52.011047 3871 server_base.cc:1034] running on GCE node
I20260924 00:49:52.011919 3871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:52.013523 3871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:52.014739 3871 hybrid_clock.cc:648] HybridClock initialized: now 1790210992014690 us; error 44 us; skew 500 ppm
I20260924 00:49:52.015101 3871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:52 dist-test-slave-rtvz krb5kdc[3857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:52.018285 3871 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:49:52.019779 3871 webserver.cc:460] Webserver started at http://127.0.222.62:42517/ using document root <none> and password file <none>
I20260924 00:49:52.020457 3871 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:52.020560 3871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:52.020812 3871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:52.023844 3871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "34e6d1508db14b15896e20d21409fba6"
format_stamp: "Formatted at 2026-09-24 00:49:52 on dist-test-slave-rtvz"
server_key: "9b6f2816e294dcecd3ddf6080e37b4d0"
server_key_iv: "8b55cd4c6a94cc601febad0af66f6731"
server_key_version: "encryptionkey@0"
I20260924 00:49:52.024595 3871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "34e6d1508db14b15896e20d21409fba6"
format_stamp: "Formatted at 2026-09-24 00:49:52 on dist-test-slave-rtvz"
server_key: "9b6f2816e294dcecd3ddf6080e37b4d0"
server_key_iv: "8b55cd4c6a94cc601febad0af66f6731"
server_key_version: "encryptionkey@0"
I20260924 00:49:52.029798 3871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:49:52.033162 3886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:52.034592 3871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:52.034751 3871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "34e6d1508db14b15896e20d21409fba6"
format_stamp: "Formatted at 2026-09-24 00:49:52 on dist-test-slave-rtvz"
server_key: "9b6f2816e294dcecd3ddf6080e37b4d0"
server_key_iv: "8b55cd4c6a94cc601febad0af66f6731"
server_key_version: "encryptionkey@0"
I20260924 00:49:52.034960 3871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:52.051947 3871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:52.056341 3871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:52.056615 3871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:52.074301 3871 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:39803
I20260924 00:49:52.074324 3937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:39803 every 8 connection(s)
I20260924 00:49:52.075763 3871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:49:52.080109 3938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:52.086310 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 3871
I20260924 00:49:52.086576 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:49:52.087338 888 external_mini_cluster.cc:1428] Setting key b145023cc8bef6c6f9f7dc22241d9efa
I20260924 00:49:52.091888 3938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6: Bootstrap starting.
I20260924 00:49:52.095238 3938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:52.096323 3938 log.cc:824] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:52.099368 3938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6: No bootstrap required, opened a new log
Sep 24 00:49:52 dist-test-slave-rtvz krb5kdc[3857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:52.108289 3938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e6d1508db14b15896e20d21409fba6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39803 } }
I20260924 00:49:52.108763 3938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:52.108896 3938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e6d1508db14b15896e20d21409fba6, State: Initialized, Role: FOLLOWER
I20260924 00:49:52.109620 3938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e6d1508db14b15896e20d21409fba6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39803 } }
I20260924 00:49:52.109839 3938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:49:52.109999 3938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:49:52.110162 3938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:52.110651 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:52.089590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.0.222.62:39803)
0924 00:49:52.090462 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:52.090476 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:52.090521 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:49:52.092353 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:52.092378 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:52.092827 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:52.093262 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:52.094849 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:52.096229 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:52.097827 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:52.098517 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:52.098686 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:52.103027 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:52.103035 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:52.103055 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:52.103101 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:52.106264 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:52.107099 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:52.107106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:52.107113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:52.107282 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:52.107649 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:52.107652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:52.107654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:52.108195 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:52.108421 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:52.108959 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 00:49:52.109447 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":354,"threads_started":1}
I20260924 00:49:52.112555 3938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e6d1508db14b15896e20d21409fba6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39803 } }
I20260924 00:49:52.113103 3938 leader_election.cc:302] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34e6d1508db14b15896e20d21409fba6; no voters:
I20260924 00:49:52.113765 3938 leader_election.cc:288] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:49:52.113878 3943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:52.114674 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:49:52.114931 3943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [term 1 LEADER]: Becoming Leader. State: Replica: 34e6d1508db14b15896e20d21409fba6, State: Running, Role: LEADER
I20260924 00:49:52.115482 3943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e6d1508db14b15896e20d21409fba6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39803 } }
I20260924 00:49:52.116051 3938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:49:52.118191 3945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34e6d1508db14b15896e20d21409fba6. Latest consensus state: current_term: 1 leader_uuid: "34e6d1508db14b15896e20d21409fba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e6d1508db14b15896e20d21409fba6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39803 } } }
I20260924 00:49:52.118556 3945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:52.119097 3944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34e6d1508db14b15896e20d21409fba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e6d1508db14b15896e20d21409fba6" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39803 } } }
I20260924 00:49:52.119381 3944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:52.122344 3948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:49:52.128181 3948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:49:52.139921 3948 catalog_manager.cc:1347] Generated new cluster ID: f566b92280994aafbce225c50ed08c11
I20260924 00:49:52.140028 3948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:49:52.158453 3948 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:49:52.160475 3948 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:49:52.173175 3948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34e6d1508db14b15896e20d21409fba6: Generated new TSK 0
I20260924 00:49:52.174592 3948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:49:52.186677 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:52.178833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.0.222.62:39803)
0924 00:49:52.179006 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:52.179011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:52.179541 (+ 530us) server_negotiation.cc:407] Connection header received
0924 00:49:52.179713 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:52.179718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:52.179786 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:52.179875 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:52.180925 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:52.181752 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:52.183178 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:52.183613 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:52.183734 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:52.184924 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:52.184932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:52.184937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:49:52.185836 (+ 899us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:52.185914 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:49:52.185951 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:52.186074 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:52.186191 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 00:49:52.186298 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:49:52.193430 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3871
I20260924 00:49:52.195263 3933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:52.285810 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 3871
2026-09-24T00:49:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1827 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:52 dist-test-slave-rtvz krb5kdc[3973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:52 dist-test-slave-rtvz krb5kdc[3973](info): set up 2 sockets
Sep 24 00:49:52 dist-test-slave-rtvz krb5kdc[3973](info): commencing operation
krb5kdc: starting...
W20260924 00:49:53.301604 888 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:49:53 dist-test-slave-rtvz krb5kdc[3973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:49:53.485172 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38857
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:38857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:53.800405 3986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:53.800707 3986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:53.800828 3986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:53.808261 3986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:49:53.808386 3986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:53.808468 3986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:53.808542 3986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:49:53.808609 3986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:49:53.818846 3986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:38857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:53.820218 3986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:53.821691 3986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:53.833751 3991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:53.833631 3992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:53.833660 3994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:53.834254 3986 server_base.cc:1034] running on GCE node
I20260924 00:49:53.835068 3986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:53.836488 3986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:53.837709 3986 hybrid_clock.cc:648] HybridClock initialized: now 1790210993837651 us; error 42 us; skew 500 ppm
I20260924 00:49:53.838109 3986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:53 dist-test-slave-rtvz krb5kdc[3973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:53.841482 3986 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:49:53.843219 3986 webserver.cc:460] Webserver started at http://127.0.222.62:41299/ using document root <none> and password file <none>
I20260924 00:49:53.843968 3986 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:53.844095 3986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:53.844419 3986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:53.847625 3986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "eeecb5e973994a1cad35068296eff6de"
format_stamp: "Formatted at 2026-09-24 00:49:53 on dist-test-slave-rtvz"
server_key: "a73aeee27f638d1a54cb0cde153ebd84"
server_key_iv: "5aeb5e51cdc8ed565a8cba9abc0687b1"
server_key_version: "encryptionkey@0"
I20260924 00:49:53.848415 3986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "eeecb5e973994a1cad35068296eff6de"
format_stamp: "Formatted at 2026-09-24 00:49:53 on dist-test-slave-rtvz"
server_key: "a73aeee27f638d1a54cb0cde153ebd84"
server_key_iv: "5aeb5e51cdc8ed565a8cba9abc0687b1"
server_key_version: "encryptionkey@0"
I20260924 00:49:53.853906 3986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:49:53.857720 4002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:53.859213 3986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:49:53.859400 3986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "eeecb5e973994a1cad35068296eff6de"
format_stamp: "Formatted at 2026-09-24 00:49:53 on dist-test-slave-rtvz"
server_key: "a73aeee27f638d1a54cb0cde153ebd84"
server_key_iv: "5aeb5e51cdc8ed565a8cba9abc0687b1"
server_key_version: "encryptionkey@0"
I20260924 00:49:53.859604 3986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:53.890039 3986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:53.895013 3986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:53.895331 3986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:53.915162 3986 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:38857
I20260924 00:49:53.915194 4055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:38857 every 8 connection(s)
I20260924 00:49:53.916726 3986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:49:53.921003 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:49:53.923594 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 3986
I20260924 00:49:53.923846 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:49:53.924530 888 external_mini_cluster.cc:1428] Setting key 8d10c4c85549a7307ee126f43f1497ae
I20260924 00:49:53.934254 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de: Bootstrap starting.
I20260924 00:49:53.937994 4056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:53.939023 4056 log.cc:824] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de: Log is configured to *not* fsync() on all Append() calls
Sep 24 00:49:53 dist-test-slave-rtvz krb5kdc[3973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:53.942245 4056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de: No bootstrap required, opened a new log
I20260924 00:49:53.949738 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:53.926791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.0.222.62:38857)
0924 00:49:53.927599 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:53.927616 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:53.927666 (+ 50us) server_negotiation.cc:407] Connection header received
0924 00:49:53.929864 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:53.929898 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:53.930508 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:53.931025 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:53.932573 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:53.934621 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:53.935974 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:53.937101 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:53.937320 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:53.941546 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:53.941554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:53.941571 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:53.941611 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:53.944897 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:53.945727 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:53.945734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:53.945742 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:53.945962 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:53.946358 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:53.946365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:53.946367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:53.947008 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:53.947454 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:53.947955 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 00:49:53.948514 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":339,"threads_started":1}
I20260924 00:49:53.950769 4056 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeecb5e973994a1cad35068296eff6de" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38857 } }
I20260924 00:49:53.951175 4056 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:53.951288 4056 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeecb5e973994a1cad35068296eff6de, State: Initialized, Role: FOLLOWER
I20260924 00:49:53.951869 4056 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeecb5e973994a1cad35068296eff6de" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38857 } }
I20260924 00:49:53.952097 4056 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:49:53.952221 4056 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:49:53.952378 4056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:53.954859 4056 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeecb5e973994a1cad35068296eff6de" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38857 } }
I20260924 00:49:53.955394 4056 leader_election.cc:302] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeecb5e973994a1cad35068296eff6de; no voters:
I20260924 00:49:53.956195 4056 leader_election.cc:288] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:49:53.956259 4061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:53.957037 4061 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [term 1 LEADER]: Becoming Leader. State: Replica: eeecb5e973994a1cad35068296eff6de, State: Running, Role: LEADER
I20260924 00:49:53.957579 4061 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeecb5e973994a1cad35068296eff6de" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38857 } }
I20260924 00:49:53.959415 4061 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeecb5e973994a1cad35068296eff6de. Latest consensus state: current_term: 1 leader_uuid: "eeecb5e973994a1cad35068296eff6de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeecb5e973994a1cad35068296eff6de" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38857 } } }
I20260924 00:49:53.959895 4061 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:53.961995 4063 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeecb5e973994a1cad35068296eff6de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeecb5e973994a1cad35068296eff6de" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 38857 } } }
I20260924 00:49:53.962404 4063 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:53.963579 4056 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:49:53.963775 4065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:49:53.969338 4065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:49:53.978850 4065 catalog_manager.cc:1347] Generated new cluster ID: 19f401a0d37a4be7b3536534f428b03e
I20260924 00:49:53.978989 4065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:49:53.998287 4065 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:49:53.999787 4065 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:49:54.008311 4065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeecb5e973994a1cad35068296eff6de: Generated new TSK 0
I20260924 00:49:54.009289 4065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:49:54.068701 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38857
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:54.395668 4082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:54.395972 4082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:54.396108 4082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:54.403515 4082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:54.403635 4082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:54.403770 4082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:49:54.413749 4082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:54.415051 4082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:54.416378 4082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:54.429558 4090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:54.429631 4088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:54.429582 4087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:54.430227 4082 server_base.cc:1034] running on GCE node
I20260924 00:49:54.430790 4082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:54.431754 4082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:54.432936 4082 hybrid_clock.cc:648] HybridClock initialized: now 1790210994432907 us; error 55 us; skew 500 ppm
I20260924 00:49:54.433271 4082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:54 dist-test-slave-rtvz krb5kdc[3973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:54.436445 4082 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:49:54.437996 4082 webserver.cc:460] Webserver started at http://127.0.222.1:40571/ using document root <none> and password file <none>
I20260924 00:49:54.438680 4082 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:54.438800 4082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:54.439090 4082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:54.442139 4082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "02c39d68d62d4fada5288d8cce1b451e"
format_stamp: "Formatted at 2026-09-24 00:49:54 on dist-test-slave-rtvz"
server_key: "b9a6169b48bb0a7b48a8938881bab29e"
server_key_iv: "fe22f4b872a533a2c2b69ddfbf91171d"
server_key_version: "encryptionkey@0"
I20260924 00:49:54.442922 4082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "02c39d68d62d4fada5288d8cce1b451e"
format_stamp: "Formatted at 2026-09-24 00:49:54 on dist-test-slave-rtvz"
server_key: "b9a6169b48bb0a7b48a8938881bab29e"
server_key_iv: "fe22f4b872a533a2c2b69ddfbf91171d"
server_key_version: "encryptionkey@0"
I20260924 00:49:54.448478 4082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:49:54.452133 4097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:54.453622 4082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:49:54.453789 4082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "02c39d68d62d4fada5288d8cce1b451e"
format_stamp: "Formatted at 2026-09-24 00:49:54 on dist-test-slave-rtvz"
server_key: "b9a6169b48bb0a7b48a8938881bab29e"
server_key_iv: "fe22f4b872a533a2c2b69ddfbf91171d"
server_key_version: "encryptionkey@0"
I20260924 00:49:54.453989 4082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:54.470647 4082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:54.474841 4082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:54.475127 4082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:54.476142 4082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:54.477615 4082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:54.477710 4082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:54.477813 4082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:54.477875 4082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:54.506091 4082 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44059
I20260924 00:49:54.506112 4211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44059 every 8 connection(s)
I20260924 00:49:54.507458 4082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:49:54.508591 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 4082
I20260924 00:49:54.508827 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:49:54.509614 888 external_mini_cluster.cc:1428] Setting key 938c3cb1629120516282b9a2ab9098b4
Sep 24 00:49:54 dist-test-slave-rtvz krb5kdc[3973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:49:54.528790 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:54.510246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:40973 (local address 127.0.222.62:38857)
0924 00:49:54.510704 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:54.510710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:54.511865 (+ 1155us) server_negotiation.cc:407] Connection header received
0924 00:49:54.513740 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:54.513746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:54.513847 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:54.513962 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:54.515467 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:54.516424 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:54.518325 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:54.518914 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:54.519035 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:54.522533 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:54.522539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:54.522543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:54.522582 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:54.525874 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:54.526610 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:54.526616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:54.526620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:54.526766 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:54.527165 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:54.527170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:54.527174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:54.527523 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:54.527644 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:54.528363 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 00:49:54.528541 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260924 00:49:54.530009 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:54.510912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38857 (local address 127.0.222.1:40973)
0924 00:49:54.511706 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:54.511751 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:54.513468 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:54.514173 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:54.514181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:54.514707 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:54.515288 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:54.515305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:54.516591 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:54.516596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:54.518175 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:54.518187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:54.519125 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:54.519130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:54.519345 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:54.519968 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:54.519990 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:54.522288 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:54.526015 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:54.526023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:54.526038 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:54.526471 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:54.526870 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:54.526873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:54.526875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:54.527053 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:54.527629 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:54.527634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:54.527991 (+ 357us) client_negotiation.cc:769] Sending connection context
0924 00:49:54.528242 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 00:49:54.528641 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":321,"threads_started":1}
I20260924 00:49:54.531626 4212 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38857
I20260924 00:49:54.531906 4212 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:54.532642 4212 heartbeater.cc:511] Master 127.0.222.62:38857 requested a full tablet report, sending...
I20260924 00:49:54.534674 4020 ts_manager.cc:194] Registered new tserver with Master: 02c39d68d62d4fada5288d8cce1b451e (127.0.222.1:44059)
I20260924 00:49:54.536144 4020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:40973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:49:54.552618 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38857
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:54.873853 4218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:54.874159 4218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:54.874305 4218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:54.881820 4218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:54.881954 4218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:54.882110 4218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:49:54.892056 4218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:54.893430 4218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:54.894719 4218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:54.907434 4224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:54.907467 4226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:54.907395 4223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:54.908414 4218 server_base.cc:1034] running on GCE node
I20260924 00:49:54.909092 4218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:54.910130 4218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:54.911355 4218 hybrid_clock.cc:648] HybridClock initialized: now 1790210994911294 us; error 55 us; skew 500 ppm
I20260924 00:49:54.911669 4218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:54 dist-test-slave-rtvz krb5kdc[3973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:54.915354 4218 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:49:54.917452 4218 webserver.cc:460] Webserver started at http://127.0.222.2:45873/ using document root <none> and password file <none>
I20260924 00:49:54.918380 4218 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:54.918524 4218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:54.918850 4218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:54.922855 4218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "c4d5458ac82841439106bac86d64adb9"
format_stamp: "Formatted at 2026-09-24 00:49:54 on dist-test-slave-rtvz"
server_key: "499c192e04e46102ea7376056fb8f24f"
server_key_iv: "eb4e990947ee493d258685d97ce0dea5"
server_key_version: "encryptionkey@0"
I20260924 00:49:54.923866 4218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "c4d5458ac82841439106bac86d64adb9"
format_stamp: "Formatted at 2026-09-24 00:49:54 on dist-test-slave-rtvz"
server_key: "499c192e04e46102ea7376056fb8f24f"
server_key_iv: "eb4e990947ee493d258685d97ce0dea5"
server_key_version: "encryptionkey@0"
I20260924 00:49:54.930724 4218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 00:49:54.935443 4233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:54.936975 4218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:49:54.937145 4218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "c4d5458ac82841439106bac86d64adb9"
format_stamp: "Formatted at 2026-09-24 00:49:54 on dist-test-slave-rtvz"
server_key: "499c192e04e46102ea7376056fb8f24f"
server_key_iv: "eb4e990947ee493d258685d97ce0dea5"
server_key_version: "encryptionkey@0"
I20260924 00:49:54.937390 4218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:54.960596 4218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:54.965344 4218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:54.965637 4218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:54.966696 4218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:54.968168 4218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:54.968250 4218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:54.968370 4218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:54.968423 4218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:54.998153 4218 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:38229
I20260924 00:49:54.998204 4346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:38229 every 8 connection(s)
I20260924 00:49:54.999465 4218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:49:55.000814 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 4218
I20260924 00:49:55.001009 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:49:55.001508 888 external_mini_cluster.cc:1428] Setting key 63b633042ece4b28c0595c2f4592d865
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[3973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:49:55.023713 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:55.002353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45521 (local address 127.0.222.62:38857)
0924 00:49:55.002547 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:55.002553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:55.004079 (+ 1526us) server_negotiation.cc:407] Connection header received
0924 00:49:55.007003 (+ 2924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:55.007009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:55.007096 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:55.007190 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:55.009443 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.010236 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.011962 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.012557 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.012684 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:55.016718 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:55.016726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:55.016730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:55.016778 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:55.020182 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.021142 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.021147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.021149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.021320 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.021803 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.021809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.021812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.022118 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:55.022258 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:55.023083 (+ 825us) server_negotiation.cc:299] Negotiation successful
0924 00:49:55.023372 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:49:55.024777 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:55.002922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38857 (local address 127.0.222.2:45521)
0924 00:49:55.003896 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:55.003951 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:55.006622 (+ 2671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:55.007497 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:55.007510 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:55.008302 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:55.009191 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:55.009272 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.010409 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.010413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:55.011798 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:55.011811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.012799 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.012804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:55.013075 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:55.013797 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:55.013827 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:55.016389 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:55.020374 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:55.020382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:55.020398 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:55.021007 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:55.021422 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:55.021425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:55.021428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:55.021670 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:55.022252 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:55.022261 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:55.022659 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 00:49:55.022959 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 00:49:55.023616 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":391,"threads_started":1}
I20260924 00:49:55.027879 4347 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38857
I20260924 00:49:55.028678 4347 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:55.030714 4347 heartbeater.cc:511] Master 127.0.222.62:38857 requested a full tablet report, sending...
I20260924 00:49:55.035108 4020 ts_manager.cc:194] Registered new tserver with Master: c4d5458ac82841439106bac86d64adb9 (127.0.222.2:38229)
I20260924 00:49:55.037623 4020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:45521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:49:55.066766 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38857
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:55.396512 4352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:55.396811 4352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:55.396919 4352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:55.406409 4352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:55.406560 4352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:55.406752 4352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:49:55.417675 4352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:38857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:55.419005 4352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:55.420442 4352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:55.433307 4357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:55.433457 4360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:55.433300 4358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:55.434310 4352 server_base.cc:1034] running on GCE node
I20260924 00:49:55.435014 4352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:55.436026 4352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:55.437248 4352 hybrid_clock.cc:648] HybridClock initialized: now 1790210995437173 us; error 42 us; skew 500 ppm
I20260924 00:49:55.437639 4352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[3973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:55.440871 4352 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:49:55.442437 4352 webserver.cc:460] Webserver started at http://127.0.222.3:39493/ using document root <none> and password file <none>
I20260924 00:49:55.443121 4352 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:55.443217 4352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:55.443475 4352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:55.446489 4352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "1fabd9d8a04145b494aeff6ac3655fa9"
format_stamp: "Formatted at 2026-09-24 00:49:55 on dist-test-slave-rtvz"
server_key: "ac0343208b8f24253286a8ab94d4c014"
server_key_iv: "3166defabaafe2797aa6baf7d5b3f9ea"
server_key_version: "encryptionkey@0"
I20260924 00:49:55.447229 4352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "1fabd9d8a04145b494aeff6ac3655fa9"
format_stamp: "Formatted at 2026-09-24 00:49:55 on dist-test-slave-rtvz"
server_key: "ac0343208b8f24253286a8ab94d4c014"
server_key_iv: "3166defabaafe2797aa6baf7d5b3f9ea"
server_key_version: "encryptionkey@0"
I20260924 00:49:55.452687 4352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 00:49:55.456463 4367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:55.458084 4352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:49:55.458266 4352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "1fabd9d8a04145b494aeff6ac3655fa9"
format_stamp: "Formatted at 2026-09-24 00:49:55 on dist-test-slave-rtvz"
server_key: "ac0343208b8f24253286a8ab94d4c014"
server_key_iv: "3166defabaafe2797aa6baf7d5b3f9ea"
server_key_version: "encryptionkey@0"
I20260924 00:49:55.458477 4352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:55.475853 4352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:55.480173 4352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:55.480460 4352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:55.481473 4352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:55.482837 4352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:55.482915 4352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:55.483027 4352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:55.483080 4352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:55.518148 4352 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:44383
I20260924 00:49:55.518165 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:44383 every 8 connection(s)
I20260924 00:49:55.519413 4352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:49:55.521896 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 4352
I20260924 00:49:55.522104 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:49:55.522692 888 external_mini_cluster.cc:1428] Setting key 8629690aa1a50e0f18ac8281befeea3e
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[3973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:55.539018 4212 heartbeater.cc:503] Master 127.0.222.62:38857 was elected leader, sending a full tablet report...
I20260924 00:49:55.546084 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:55.522255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:39257 (local address 127.0.222.62:38857)
0924 00:49:55.522459 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:55.522463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:55.523978 (+ 1515us) server_negotiation.cc:407] Connection header received
0924 00:49:55.526137 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:55.526141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:55.526239 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:55.526314 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:55.527855 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.528651 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.530426 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.530868 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.530991 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:55.539967 (+ 8976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:55.539972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:55.539976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:55.540010 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:55.542900 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.543663 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.543668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.543670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.543805 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.544268 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.544272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.544274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.544570 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:55.544697 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:55.545478 (+ 781us) server_negotiation.cc:299] Negotiation successful
0924 00:49:55.545705 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:49:55.547096 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:55.522953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:38857 (local address 127.0.222.3:39257)
0924 00:49:55.523803 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:55.523861 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:55.525919 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:55.526601 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:55.526612 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:55.527113 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:55.527705 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:55.527724 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.528798 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.528802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:55.530265 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:55.530278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.531088 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.531093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:55.531315 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:55.537042 (+ 5727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:55.537071 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:55.539730 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:55.543047 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:55.543057 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:55.543077 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:55.543536 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:55.543922 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:55.543928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:55.543932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:55.544154 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:55.544695 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:55.544704 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:55.545070 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 00:49:55.545349 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 00:49:55.545942 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":361,"threads_started":1}
I20260924 00:49:55.548501 4480 heartbeater.cc:347] Connected to a master server at 127.0.222.62:38857
I20260924 00:49:55.548816 4480 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:55.549656 4480 heartbeater.cc:511] Master 127.0.222.62:38857 requested a full tablet report, sending...
I20260924 00:49:55.551267 4020 ts_manager.cc:194] Registered new tserver with Master: 1fabd9d8a04145b494aeff6ac3655fa9 (127.0.222.3:44383)
I20260924 00:49:55.552120 4020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:39257
I20260924 00:49:55.563522 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[3973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210995, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[3973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210995, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:55.611617 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:55.596281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.0.222.62:38857)
0924 00:49:55.596463 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:55.596467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:55.596964 (+ 497us) server_negotiation.cc:407] Connection header received
0924 00:49:55.597093 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:55.597097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:55.597167 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:55.597297 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:55.598952 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.599738 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.601023 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.601532 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.601635 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:55.605781 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:55.605787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:55.605790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:55.605831 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:55.608632 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.609506 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.609511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.609514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.609707 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.610271 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.610276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.610278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.610667 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:55.610876 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:55.611172 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 00:49:55.611363 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 00:49:55.612178 4020 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45916
I20260924 00:49:55.613983 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4082
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[3973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210995, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:49:55.630849 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:55.615094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.0.222.1:44059)
0924 00:49:55.615718 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:55.615725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:55.615753 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:49:55.615810 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:55.615817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:55.616044 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:55.616174 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:55.617505 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.618606 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.619900 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.620508 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.620613 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:55.624937 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:55.624954 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:55.624971 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:55.625012 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:55.628061 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.628730 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.628738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.628740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.628867 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.629325 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.629329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.629331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.629702 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:55.629962 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:55.630207 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 00:49:55.630448 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":333,"threads_started":1}
W20260924 00:49:55.632253 4207 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47368
W20260924 00:49:55.633384 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4082: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:55.633587 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 4082
I20260924 00:49:55.646240 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4218
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[3973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210995, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:49:55.663563 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:55.647193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48524 (local address 127.0.222.2:38229)
0924 00:49:55.647830 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:55.647838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:55.647866 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:49:55.647934 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:55.647941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:55.648160 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:55.648324 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:55.649681 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.650879 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.652115 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.652676 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.652799 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:55.657105 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:55.657119 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:55.657134 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:55.657221 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:55.660284 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.661028 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.661036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.661039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.661172 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.661674 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.661677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.661679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.662141 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:55.662465 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:55.662851 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 00:49:55.663204 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":255,"threads_started":1}
W20260924 00:49:55.665091 4342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48524
W20260924 00:49:55.666390 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:55.666585 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 4218
I20260924 00:49:55.678893 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4352
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[3973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210995, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:49:55.695474 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:55.679759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.0.222.3:44383)
0924 00:49:55.680464 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:55.680471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:55.680500 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:49:55.680559 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:55.680566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:55.680770 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:55.680905 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:55.682167 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.683198 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.684580 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:55.685153 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:55.685304 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:55.689438 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:55.689459 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:55.689476 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:55.689512 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:55.692467 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.693304 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.693317 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.693320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.693462 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:55.693854 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:55.693859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:55.693862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:55.694230 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:55.694543 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:55.694828 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 00:49:55.695134 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":399,"threads_started":1}
W20260924 00:49:55.696722 4475 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53924
W20260924 00:49:55.697907 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4352: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:49:55.698071 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 4352
I20260924 00:49:55.710088 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 3986
I20260924 00:49:55.711439 4051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:55.817559 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 3986
2026-09-24T00:49:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3536 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[4498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[4498](info): set up 2 sockets
Sep 24 00:49:55 dist-test-slave-rtvz krb5kdc[4498](info): commencing operation
krb5kdc: starting...
W20260924 00:49:56.817323 888 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:49:56 dist-test-slave-rtvz krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:49:57.009231 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41523
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:57.332168 4511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:57.332466 4511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:57.332582 4511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:57.339903 4511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:49:57.340018 4511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:57.340068 4511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:57.340144 4511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:49:57.340198 4511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:49:57.350248 4511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:57.351549 4511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:57.352835 4511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:57.364130 4519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:57.364033 4517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:57.364166 4516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:57.365216 4511 server_base.cc:1034] running on GCE node
I20260924 00:49:57.366097 4511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:57.367535 4511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:57.368752 4511 hybrid_clock.cc:648] HybridClock initialized: now 1790210997368702 us; error 32 us; skew 500 ppm
I20260924 00:49:57.369158 4511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:57 dist-test-slave-rtvz krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:57.372355 4511 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:49:57.374073 4511 webserver.cc:460] Webserver started at http://127.0.222.62:41133/ using document root <none> and password file <none>
I20260924 00:49:57.374778 4511 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:57.374907 4511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:57.375204 4511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:57.378304 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "fe1c2856e513474090997d6bd3453f2a"
format_stamp: "Formatted at 2026-09-24 00:49:57 on dist-test-slave-rtvz"
server_key: "b2f58c88c5d347329edd21a43b7b0202"
server_key_iv: "59ce210d4ec3405fc65dd9efcb1d223e"
server_key_version: "encryptionkey@0"
I20260924 00:49:57.379130 4511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "fe1c2856e513474090997d6bd3453f2a"
format_stamp: "Formatted at 2026-09-24 00:49:57 on dist-test-slave-rtvz"
server_key: "b2f58c88c5d347329edd21a43b7b0202"
server_key_iv: "59ce210d4ec3405fc65dd9efcb1d223e"
server_key_version: "encryptionkey@0"
I20260924 00:49:57.384656 4511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:49:57.388437 4526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:57.389999 4511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:49:57.390177 4511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "fe1c2856e513474090997d6bd3453f2a"
format_stamp: "Formatted at 2026-09-24 00:49:57 on dist-test-slave-rtvz"
server_key: "b2f58c88c5d347329edd21a43b7b0202"
server_key_iv: "59ce210d4ec3405fc65dd9efcb1d223e"
server_key_version: "encryptionkey@0"
I20260924 00:49:57.390400 4511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:57.409741 4511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:57.414682 4511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:57.415206 4511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:57.434304 4511 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41523
I20260924 00:49:57.434335 4577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41523 every 8 connection(s)
I20260924 00:49:57.435691 4511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:49:57.437743 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 4511
I20260924 00:49:57.437966 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:49:57.438820 888 external_mini_cluster.cc:1428] Setting key 98dfa6a2eff96d18b4f70b8e11512828
I20260924 00:49:57.440742 4578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 00:49:57 dist-test-slave-rtvz krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:57.454900 4578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a: Bootstrap starting.
I20260924 00:49:57.458221 4578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a: Neither blocks nor log segments found. Creating new log.
I20260924 00:49:57.459363 4578 log.cc:824] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:49:57.462836 4578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a: No bootstrap required, opened a new log
I20260924 00:49:57.463639 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:57.441024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.0.222.62:41523)
0924 00:49:57.441863 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:57.441875 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:57.441918 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:49:57.443598 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:57.443622 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:57.444093 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:57.444464 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:57.446479 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:57.448016 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:57.449764 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:57.450499 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:57.450645 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:57.455522 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:57.455532 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:57.455548 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:57.455594 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:57.458970 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:57.459881 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:57.459892 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:57.459902 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:57.460173 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:57.460576 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:57.460586 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:57.460589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:57.461294 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:57.461525 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:57.462065 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 00:49:57.462482 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":403,"threads_started":1}
I20260924 00:49:57.471038 4578 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1c2856e513474090997d6bd3453f2a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41523 } }
I20260924 00:49:57.471419 4578 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:49:57.471514 4578 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1c2856e513474090997d6bd3453f2a, State: Initialized, Role: FOLLOWER
I20260924 00:49:57.472219 4578 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1c2856e513474090997d6bd3453f2a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41523 } }
I20260924 00:49:57.472473 4578 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:49:57.472618 4578 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:49:57.472793 4578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:49:57.475020 4578 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1c2856e513474090997d6bd3453f2a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41523 } }
I20260924 00:49:57.475531 4578 leader_election.cc:302] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1c2856e513474090997d6bd3453f2a; no voters:
I20260924 00:49:57.476172 4578 leader_election.cc:288] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:49:57.476290 4583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:49:57.477010 4583 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [term 1 LEADER]: Becoming Leader. State: Replica: fe1c2856e513474090997d6bd3453f2a, State: Running, Role: LEADER
I20260924 00:49:57.477533 4583 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1c2856e513474090997d6bd3453f2a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41523 } }
I20260924 00:49:57.478259 4578 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:49:57.480276 4585 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1c2856e513474090997d6bd3453f2a. Latest consensus state: current_term: 1 leader_uuid: "fe1c2856e513474090997d6bd3453f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1c2856e513474090997d6bd3453f2a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41523 } } }
I20260924 00:49:57.480669 4585 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:57.481469 4584 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1c2856e513474090997d6bd3453f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1c2856e513474090997d6bd3453f2a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41523 } } }
I20260924 00:49:57.481899 4584 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a [sys.catalog]: This master's current role is: LEADER
I20260924 00:49:57.482933 4589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:49:57.488752 4589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:49:57.499516 4589 catalog_manager.cc:1347] Generated new cluster ID: 528b64625f6c4d1d9ee4ff0de545ca71
I20260924 00:49:57.499646 4589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:49:57.525311 4589 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:49:57.527061 4589 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:49:57.539055 4589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1c2856e513474090997d6bd3453f2a: Generated new TSK 0
I20260924 00:49:57.540022 4589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:49:57.607118 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41523
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:57.937947 4604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:57.938261 4604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:57.938375 4604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:57.945902 4604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:57.946033 4604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:57.946180 4604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:49:57.956938 4604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:57.958372 4604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:57.959776 4604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:57.974088 4609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:57.974195 4612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:57.974112 4610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:57.974922 4604 server_base.cc:1034] running on GCE node
I20260924 00:49:57.975590 4604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:57.976625 4604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:57.977845 4604 hybrid_clock.cc:648] HybridClock initialized: now 1790210997977778 us; error 44 us; skew 500 ppm
I20260924 00:49:57.978266 4604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:57 dist-test-slave-rtvz krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:57.981493 4604 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:49:57.983146 4604 webserver.cc:460] Webserver started at http://127.0.222.1:45357/ using document root <none> and password file <none>
I20260924 00:49:57.983842 4604 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:57.983966 4604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:57.984263 4604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:57.987521 4604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "046a7563542d4f96861f698cb084d0bf"
format_stamp: "Formatted at 2026-09-24 00:49:57 on dist-test-slave-rtvz"
server_key: "7b411612f178d714a70990a28b1a7a5c"
server_key_iv: "86ef0279f1986267f80542b81c59b98a"
server_key_version: "encryptionkey@0"
I20260924 00:49:57.988277 4604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "046a7563542d4f96861f698cb084d0bf"
format_stamp: "Formatted at 2026-09-24 00:49:57 on dist-test-slave-rtvz"
server_key: "7b411612f178d714a70990a28b1a7a5c"
server_key_iv: "86ef0279f1986267f80542b81c59b98a"
server_key_version: "encryptionkey@0"
I20260924 00:49:57.993839 4604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:49:57.997678 4619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:57.999385 4604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:49:57.999571 4604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "046a7563542d4f96861f698cb084d0bf"
format_stamp: "Formatted at 2026-09-24 00:49:57 on dist-test-slave-rtvz"
server_key: "7b411612f178d714a70990a28b1a7a5c"
server_key_iv: "86ef0279f1986267f80542b81c59b98a"
server_key_version: "encryptionkey@0"
I20260924 00:49:57.999883 4604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:58.023078 4604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:58.027410 4604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:58.027679 4604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:58.028658 4604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:58.030200 4604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:58.030285 4604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:58.030408 4604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:58.030462 4604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:58.062132 4604 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:37331
I20260924 00:49:58.062213 4732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:37331 every 8 connection(s)
I20260924 00:49:58.063529 4604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:49:58.067737 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 4604
I20260924 00:49:58.068022 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:49:58.068657 888 external_mini_cluster.cc:1428] Setting key 516b3c38db52fd3e8d23ba88a1305076
Sep 24 00:49:58 dist-test-slave-rtvz krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:58.086740 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:58.066286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:56027 (local address 127.0.222.62:41523)
0924 00:49:58.066738 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:58.066743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:58.068013 (+ 1270us) server_negotiation.cc:407] Connection header received
0924 00:49:58.070003 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:58.070008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:58.070073 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:58.070170 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:58.071960 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:58.073090 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:58.075013 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:58.075569 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:58.075709 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:58.079899 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:58.079904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:58.079908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:58.079969 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:58.082758 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:58.083911 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:58.083916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:58.083918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:58.084061 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:58.084658 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:58.084662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:58.084664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:58.084977 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:58.085096 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:58.086184 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0924 00:49:58.086454 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:49:58.088140 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:58.066757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41523 (local address 127.0.222.1:56027)
0924 00:49:58.067667 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:58.067723 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:58.069690 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:58.070452 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:58.070465 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:58.071134 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:58.071750 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:58.071770 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:58.073333 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:58.073338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:58.074838 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:58.074853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:58.075829 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:58.075835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:58.076089 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:58.076761 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:58.076786 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:58.079581 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:58.082992 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:58.083004 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:58.083028 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:58.083734 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:58.084184 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:58.084190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:58.084193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:58.084494 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:58.085122 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:58.085131 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:58.085682 (+ 551us) client_negotiation.cc:769] Sending connection context
0924 00:49:58.086064 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 00:49:58.086696 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":364,"threads_started":1}
I20260924 00:49:58.089967 4733 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260924 00:49:58.090323 4733 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:58.091135 4733 heartbeater.cc:511] Master 127.0.222.62:41523 requested a full tablet report, sending...
I20260924 00:49:58.093787 4543 ts_manager.cc:194] Registered new tserver with Master: 046a7563542d4f96861f698cb084d0bf (127.0.222.1:37331)
I20260924 00:49:58.095801 4543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:56027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:49:58.114682 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41523
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:58.439950 4739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:58.440231 4739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:58.440356 4739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:58.447768 4739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:58.447903 4739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:58.448061 4739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:49:58.458618 4739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:58.459970 4739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:58.461364 4739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:58.477713 4744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:58.477665 4747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:58.477638 4745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:58.478653 4739 server_base.cc:1034] running on GCE node
I20260924 00:49:58.479332 4739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:58.480289 4739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:58.481477 4739 hybrid_clock.cc:648] HybridClock initialized: now 1790210998481426 us; error 67 us; skew 500 ppm
I20260924 00:49:58.481786 4739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:58 dist-test-slave-rtvz krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:58.484791 4739 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:49:58.486308 4739 webserver.cc:460] Webserver started at http://127.0.222.2:36941/ using document root <none> and password file <none>
I20260924 00:49:58.486968 4739 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:58.487071 4739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:58.487344 4739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:58.490309 4739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "926e198dfcc94680b1391b9cfdabaa92"
format_stamp: "Formatted at 2026-09-24 00:49:58 on dist-test-slave-rtvz"
server_key: "36411575550d80cc507a9d02cf61e1b5"
server_key_iv: "d7449b19bdce3f5f7b3b71fde3f4fc8e"
server_key_version: "encryptionkey@0"
I20260924 00:49:58.491041 4739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "926e198dfcc94680b1391b9cfdabaa92"
format_stamp: "Formatted at 2026-09-24 00:49:58 on dist-test-slave-rtvz"
server_key: "36411575550d80cc507a9d02cf61e1b5"
server_key_iv: "d7449b19bdce3f5f7b3b71fde3f4fc8e"
server_key_version: "encryptionkey@0"
I20260924 00:49:58.496266 4739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:49:58.499801 4754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:58.501454 4739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:49:58.501611 4739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "926e198dfcc94680b1391b9cfdabaa92"
format_stamp: "Formatted at 2026-09-24 00:49:58 on dist-test-slave-rtvz"
server_key: "36411575550d80cc507a9d02cf61e1b5"
server_key_iv: "d7449b19bdce3f5f7b3b71fde3f4fc8e"
server_key_version: "encryptionkey@0"
I20260924 00:49:58.501808 4739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:58.532434 4739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:58.537014 4739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:58.537359 4739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:58.538362 4739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:58.539855 4739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:58.539932 4739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:58.540045 4739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:58.540095 4739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:58.572199 4739 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:44251
I20260924 00:49:58.572242 4868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:44251 every 8 connection(s)
I20260924 00:49:58.573658 4739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:49:58.583529 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 4739
I20260924 00:49:58.583678 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:49:58.584308 888 external_mini_cluster.cc:1428] Setting key 1c6b3f5f7f27aae67a50b728e54bcb9f
Sep 24 00:49:58 dist-test-slave-rtvz krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:58.595920 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:58.576335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:57537 (local address 127.0.222.62:41523)
0924 00:49:58.576556 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:58.576563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:58.577855 (+ 1292us) server_negotiation.cc:407] Connection header received
0924 00:49:58.579748 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:58.579752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:58.579830 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:58.579918 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:58.581618 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:58.582396 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:58.584086 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:58.584717 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:58.584850 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:58.589153 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:58.589159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:58.589163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:58.589258 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:58.592339 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:58.593123 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:58.593131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:58.593135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:58.593336 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:58.593853 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:58.593859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:58.593863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:58.594261 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:58.594424 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:58.595253 (+ 829us) server_negotiation.cc:299] Negotiation successful
0924 00:49:58.595513 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 00:49:58.596787 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:58.576906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41523 (local address 127.0.222.2:57537)
0924 00:49:58.577685 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:58.577732 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:58.579453 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:58.580170 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:58.580181 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:58.580757 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:58.581442 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:58.581462 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:58.582553 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:58.582557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:58.583938 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:58.583952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:58.584968 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:58.584976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:58.585282 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:58.586014 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:58.586043 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:58.588872 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:58.592508 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:58.592515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:58.592530 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:58.592973 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:58.593510 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:58.593515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:58.593519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:58.593725 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:58.594390 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:58.594396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:58.594792 (+ 396us) client_negotiation.cc:769] Sending connection context
0924 00:49:58.595115 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 00:49:58.595553 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":315,"threads_started":1}
I20260924 00:49:58.598420 4869 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41523
I20260924 00:49:58.598730 4869 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:58.599514 4869 heartbeater.cc:511] Master 127.0.222.62:41523 requested a full tablet report, sending...
I20260924 00:49:58.601179 4543 ts_manager.cc:194] Registered new tserver with Master: 926e198dfcc94680b1391b9cfdabaa92 (127.0.222.2:44251)
I20260924 00:49:58.602406 4543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:57537
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:49:58.628741 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41523
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:49:58.952193 4874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:49:58.952533 4874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:49:58.952677 4874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:49:58.960228 4874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:49:58.960382 4874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:49:58.960541 4874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:49:58.970808 4874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:49:58.972203 4874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:49:58.973711 4874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:49:58.988265 4879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:58.988298 4880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:49:58.988322 4882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:49:58.989513 4874 server_base.cc:1034] running on GCE node
I20260924 00:49:58.990159 4874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:49:58.991169 4874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:49:58.992398 4874 hybrid_clock.cc:648] HybridClock initialized: now 1790210998992337 us; error 46 us; skew 500 ppm
I20260924 00:49:58.992762 4874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:49:58 dist-test-slave-rtvz krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:49:58.996500 4874 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:49:58.998391 4874 webserver.cc:460] Webserver started at http://127.0.222.3:46117/ using document root <none> and password file <none>
I20260924 00:49:58.999217 4874 fs_manager.cc:359] Metadata directory not provided
I20260924 00:49:58.999357 4874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:49:58.999683 4874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:49:59.003033 4874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "297596f50f5145928e05d58440d19ce0"
format_stamp: "Formatted at 2026-09-24 00:49:58 on dist-test-slave-rtvz"
server_key: "86915f6eaf5aa52d07e47935ec4cc2b8"
server_key_iv: "3eb5b42939bbb64d3773f70686d509ab"
server_key_version: "encryptionkey@0"
I20260924 00:49:59.003948 4874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "297596f50f5145928e05d58440d19ce0"
format_stamp: "Formatted at 2026-09-24 00:49:58 on dist-test-slave-rtvz"
server_key: "86915f6eaf5aa52d07e47935ec4cc2b8"
server_key_iv: "3eb5b42939bbb64d3773f70686d509ab"
server_key_version: "encryptionkey@0"
I20260924 00:49:59.009958 4874 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 00:49:59.014065 4889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:59.015874 4874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 00:49:59.016067 4874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "297596f50f5145928e05d58440d19ce0"
format_stamp: "Formatted at 2026-09-24 00:49:58 on dist-test-slave-rtvz"
server_key: "86915f6eaf5aa52d07e47935ec4cc2b8"
server_key_iv: "3eb5b42939bbb64d3773f70686d509ab"
server_key_version: "encryptionkey@0"
I20260924 00:49:59.016295 4874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:49:59.031831 4874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:49:59.035833 4874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:49:59.036087 4874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:49:59.037071 4874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:49:59.038407 4874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:49:59.038478 4874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:59.038578 4874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:49:59.038618 4874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:49:59.068549 4874 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:42257
I20260924 00:49:59.068578 5003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:42257 every 8 connection(s)
I20260924 00:49:59.070027 4874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:49:59.077333 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 4874
I20260924 00:49:59.077512 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:49:59.078008 888 external_mini_cluster.cc:1428] Setting key acbb754485708f072dce531fc666e892
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:59.091631 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:59.072550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47211 (local address 127.0.222.62:41523)
0924 00:49:59.072747 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:59.072753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:59.074119 (+ 1366us) server_negotiation.cc:407] Connection header received
0924 00:49:59.076234 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:59.076238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:59.076313 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:59.076442 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:59.078253 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.079097 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.080797 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.081279 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.081391 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:59.085343 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:59.085349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:59.085352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:59.085390 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:59.088467 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.089332 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.089336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.089339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.089508 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.089953 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.089957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.089959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.090252 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:59.090363 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:59.091121 (+ 758us) server_negotiation.cc:299] Negotiation successful
0924 00:49:59.091311 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 00:49:59.092993 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:59.073087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41523 (local address 127.0.222.3:47211)
0924 00:49:59.073949 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 00:49:59.073993 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:49:59.075930 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:49:59.076658 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:49:59.076667 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:49:59.077287 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:49:59.078012 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:59.078033 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.079273 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.079278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:59.080652 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:49:59.080664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.081531 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.081539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:49:59.081809 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:59.082494 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:49:59.082518 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:49:59.085032 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:49:59.088627 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:59.088635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:59.088650 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:59.089173 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:59.089603 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:49:59.089606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:49:59.089609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:49:59.089829 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:49:59.090356 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:49:59.090362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:49:59.090733 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 00:49:59.090968 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 00:49:59.091498 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":335,"threads_started":1}
I20260924 00:49:59.094647 5004 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41523
I20260924 00:49:59.094970 5004 heartbeater.cc:464] Registering TS with master...
I20260924 00:49:59.095834 5004 heartbeater.cc:511] Master 127.0.222.62:41523 requested a full tablet report, sending...
I20260924 00:49:59.097609 4542 ts_manager.cc:194] Registered new tserver with Master: 297596f50f5145928e05d58440d19ce0 (127.0.222.3:42257)
I20260924 00:49:59.098618 4542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:47211
I20260924 00:49:59.099294 4733 heartbeater.cc:503] Master 127.0.222.62:41523 was elected leader, sending a full tablet report...
I20260924 00:49:59.106964 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:49:59.154356 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:59.137664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.0.222.1:37331)
0924 00:49:59.138418 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:59.138427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:59.138460 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:49:59.138681 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:59.138689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:59.138938 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:59.139089 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:59.140716 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.141968 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.143399 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.144050 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.144179 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:59.148373 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:59.148387 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:59.148403 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:59.148439 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:59.151317 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.152041 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.152049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.152052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.152196 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.152626 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.152629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.152632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.153062 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:59.153345 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:59.153758 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 00:49:59.154032 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":339,"threads_started":1}
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:49:59.174909 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:59.158884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.0.222.62:41523)
0924 00:49:59.159088 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:59.159094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:59.159241 (+ 147us) server_negotiation.cc:407] Connection header received
0924 00:49:59.159458 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:59.159463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:59.159537 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:59.159622 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:59.160989 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.161807 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.163336 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.163783 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.163932 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:59.168600 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:59.168605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:59.168609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:59.168650 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:59.171678 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.172596 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.172602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.172605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.172759 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.173151 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.173155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.173157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.173598 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:59.173859 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:59.174251 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 00:49:59.174469 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 00:49:59.175575 4542 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56042
I20260924 00:49:59.176810 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4604
I20260924 00:49:59.190727 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:59.178021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.0.222.1:37331)
0924 00:49:59.178184 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:59.178188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:59.178355 (+ 167us) server_negotiation.cc:407] Connection header received
0924 00:49:59.178484 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:59.178488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:59.178581 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:59.178656 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:59.179891 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.180634 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.182140 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.182560 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.182660 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:59.184950 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:59.184955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:59.184959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:59.184996 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:59.188075 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.188853 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.188857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.188860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.189017 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.189439 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.189443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.189445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.189719 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:59.189960 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:59.190269 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 00:49:59.190472 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:49:59.191402 4728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:59.274979 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 4604
I20260924 00:49:59.288729 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4739
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:49:59.304523 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:59.289679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.0.222.2:44251)
0924 00:49:59.290291 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:59.290298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:59.290324 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:49:59.290392 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:59.290399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:59.290608 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:59.290735 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:59.291896 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.292874 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.294392 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.294973 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.295082 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:59.299190 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:59.299203 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:59.299219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:59.299255 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:59.302105 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.302761 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.302768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.302770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.302912 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.303223 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.303227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.303229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.303567 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:59.303785 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:59.303999 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 00:49:59.304192 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":353,"threads_started":1}
I20260924 00:49:59.305789 4864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:59.388424 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 4739
I20260924 00:49:59.401271 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4874
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790210999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:49:59.418596 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:49:59.402271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.0.222.3:42257)
0924 00:49:59.402956 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 00:49:59.402962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:49:59.402997 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:49:59.403057 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:49:59.403063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:49:59.403262 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:49:59.403394 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:49:59.404649 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.405901 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.407168 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:49:59.407748 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:49:59.407901 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:49:59.412542 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:49:59.412556 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:49:59.412573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:49:59.412613 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:49:59.415530 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.416329 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.416338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.416340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.416472 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:49:59.416944 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:49:59.416948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:49:59.416950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:49:59.417421 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:49:59.417706 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 00:49:59.418022 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 00:49:59.418238 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":301,"threads_started":1}
I20260924 00:49:59.419950 4999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:59.501653 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 4874
I20260924 00:49:59.514261 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 4511
I20260924 00:49:59.515479 4573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:49:59.622731 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 4511
2026-09-24T00:49:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3805 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[5023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[5023](info): set up 2 sockets
Sep 24 00:49:59 dist-test-slave-rtvz krb5kdc[5023](info): commencing operation
krb5kdc: starting...
W20260924 00:50:00.669149 888 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:50:00 dist-test-slave-rtvz krb5kdc[5023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:50:00.865521 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41913
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:01.216791 5036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:01.217139 5036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:01.217298 5036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:01.224754 5036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:01.224895 5036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:01.224977 5036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:01.225051 5036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:01.225118 5036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:01.235484 5036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:01.236857 5036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:01.238250 5036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:01.250185 5041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:01.250185 5044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:01.250208 5042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:01.250810 5036 server_base.cc:1034] running on GCE node
I20260924 00:50:01.251757 5036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:01.253304 5036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:01.254523 5036 hybrid_clock.cc:648] HybridClock initialized: now 1790211001254451 us; error 61 us; skew 500 ppm
I20260924 00:50:01.254853 5036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:01 dist-test-slave-rtvz krb5kdc[5023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:01.263759 5036 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:50:01.265074 5036 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:01.265184 5036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:01.265544 5036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:01.268769 5036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "28d0947daaad44afbbbd1060a3f50fbc"
format_stamp: "Formatted at 2026-09-24 00:50:01 on dist-test-slave-rtvz"
server_key: "5bf7aaffeb5b96924a0835d2f841a40b"
server_key_iv: "c46c399eb8e921dab22ed29054ebb6c5"
server_key_version: "encryptionkey@0"
I20260924 00:50:01.269625 5036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "28d0947daaad44afbbbd1060a3f50fbc"
format_stamp: "Formatted at 2026-09-24 00:50:01 on dist-test-slave-rtvz"
server_key: "5bf7aaffeb5b96924a0835d2f841a40b"
server_key_iv: "c46c399eb8e921dab22ed29054ebb6c5"
server_key_version: "encryptionkey@0"
I20260924 00:50:01.275180 5036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 00:50:01.279029 5050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:01.280546 5036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:50:01.280721 5036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "28d0947daaad44afbbbd1060a3f50fbc"
format_stamp: "Formatted at 2026-09-24 00:50:01 on dist-test-slave-rtvz"
server_key: "5bf7aaffeb5b96924a0835d2f841a40b"
server_key_iv: "c46c399eb8e921dab22ed29054ebb6c5"
server_key_version: "encryptionkey@0"
I20260924 00:50:01.280949 5036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:01.298735 5036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:01.303025 5036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:01.303298 5036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:01.321775 5036 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41913
I20260924 00:50:01.321820 5101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41913 every 8 connection(s)
I20260924 00:50:01.322751 5036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:01.324581 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 5036
I20260924 00:50:01.324879 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:01.325642 888 external_mini_cluster.cc:1428] Setting key 71dd80d5c171bcb860221ff8d26b8e21
I20260924 00:50:01.327056 5102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:01.340559 5102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc: Bootstrap starting.
I20260924 00:50:01.343923 5102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:01.345016 5102 log.cc:824] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc: Log is configured to *not* fsync() on all Append() calls
Sep 24 00:50:01 dist-test-slave-rtvz krb5kdc[5023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:01.348429 5102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc: No bootstrap required, opened a new log
I20260924 00:50:01.354084 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:01.327804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.0.222.62:41913)
0924 00:50:01.328672 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:01.328686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:01.333965 (+ 5279us) server_negotiation.cc:407] Connection header received
0924 00:50:01.335952 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:01.335979 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:01.336424 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:01.336863 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:01.338555 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:01.339890 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:01.341344 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:01.342094 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:01.342277 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:01.346552 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:01.346562 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:01.346579 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:01.346619 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:01.349803 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:01.350514 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:01.350521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:01.350531 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:01.350710 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:01.351107 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:01.351110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:01.351112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:01.351695 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:01.351987 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:01.352507 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 00:50:01.352931 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":389,"threads_started":1}
I20260924 00:50:01.356369 5102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d0947daaad44afbbbd1060a3f50fbc" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41913 } }
I20260924 00:50:01.356737 5102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:01.356840 5102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d0947daaad44afbbbd1060a3f50fbc, State: Initialized, Role: FOLLOWER
I20260924 00:50:01.357458 5102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d0947daaad44afbbbd1060a3f50fbc" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41913 } }
I20260924 00:50:01.357697 5102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:01.357815 5102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:01.357957 5102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:01.360100 5102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d0947daaad44afbbbd1060a3f50fbc" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41913 } }
I20260924 00:50:01.360592 5102 leader_election.cc:302] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28d0947daaad44afbbbd1060a3f50fbc; no voters:
I20260924 00:50:01.361150 5102 leader_election.cc:288] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:01.361323 5107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:01.361966 5107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [term 1 LEADER]: Becoming Leader. State: Replica: 28d0947daaad44afbbbd1060a3f50fbc, State: Running, Role: LEADER
I20260924 00:50:01.362483 5107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d0947daaad44afbbbd1060a3f50fbc" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41913 } }
I20260924 00:50:01.363355 5102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:01.365671 5108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28d0947daaad44afbbbd1060a3f50fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d0947daaad44afbbbd1060a3f50fbc" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41913 } } }
I20260924 00:50:01.366104 5108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:01.365970 5109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28d0947daaad44afbbbd1060a3f50fbc. Latest consensus state: current_term: 1 leader_uuid: "28d0947daaad44afbbbd1060a3f50fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d0947daaad44afbbbd1060a3f50fbc" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41913 } } }
I20260924 00:50:01.366284 5109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:01.368538 5117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:01.372730 5117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:01.381146 5117 catalog_manager.cc:1347] Generated new cluster ID: 2bae41e4c736455284b7d0d9e1feffd2
I20260924 00:50:01.381333 5117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:01.397965 5117 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:01.400033 5117 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:01.407119 5117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc: Generated new TSK 0
I20260924 00:50:01.408038 5117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:50:01.460469 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41913
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:01.802834 5129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:01.803146 5129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:01.803254 5129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:01.810765 5129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:01.810890 5129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:01.811009 5129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:01.821161 5129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:01.822573 5129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:01.823975 5129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:01.837709 5137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:01.837841 5134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:01.837736 5135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:01.838543 5129 server_base.cc:1034] running on GCE node
I20260924 00:50:01.839196 5129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:01.840335 5129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:01.841567 5129 hybrid_clock.cc:648] HybridClock initialized: now 1790211001841506 us; error 55 us; skew 500 ppm
I20260924 00:50:01.841907 5129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:01 dist-test-slave-rtvz krb5kdc[5023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:01.845342 5129 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:50:01.846598 5129 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:01.846710 5129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:01.847018 5129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:01.850065 5129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "fbb9facbbd484c0cad7e73487c25469e"
format_stamp: "Formatted at 2026-09-24 00:50:01 on dist-test-slave-rtvz"
server_key: "7e36d4a139672c1193c8728e8a078794"
server_key_iv: "37e83bcdc9dbd5cae8b513665cae681c"
server_key_version: "encryptionkey@0"
I20260924 00:50:01.850910 5129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "fbb9facbbd484c0cad7e73487c25469e"
format_stamp: "Formatted at 2026-09-24 00:50:01 on dist-test-slave-rtvz"
server_key: "7e36d4a139672c1193c8728e8a078794"
server_key_iv: "37e83bcdc9dbd5cae8b513665cae681c"
server_key_version: "encryptionkey@0"
I20260924 00:50:01.856105 5129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:50:01.859858 5143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:01.861389 5129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 00:50:01.861564 5129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "fbb9facbbd484c0cad7e73487c25469e"
format_stamp: "Formatted at 2026-09-24 00:50:01 on dist-test-slave-rtvz"
server_key: "7e36d4a139672c1193c8728e8a078794"
server_key_iv: "37e83bcdc9dbd5cae8b513665cae681c"
server_key_version: "encryptionkey@0"
I20260924 00:50:01.861784 5129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:01.875156 5129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:01.879988 5129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:01.880262 5129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:01.881392 5129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:01.882819 5129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:01.882918 5129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:01.883021 5129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:01.883097 5129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:01.914950 5129 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:37751
I20260924 00:50:01.915036 5255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:37751 every 8 connection(s)
I20260924 00:50:01.915834 5129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:01.919976 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 5129
I20260924 00:50:01.920181 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:50:01.920851 888 external_mini_cluster.cc:1428] Setting key 541cfe8b134d063bb9e258a4a02dadbe
Sep 24 00:50:01 dist-test-slave-rtvz krb5kdc[5023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:01.936203 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:01.918422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:38983 (local address 127.0.222.62:41913)
0924 00:50:01.918913 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:01.918918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:01.919976 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 00:50:01.922028 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:01.922034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:01.922116 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:01.922220 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:01.923924 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:01.924642 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:01.926287 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:01.926706 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:01.926829 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:01.930588 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:01.930594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:01.930598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:01.930633 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:01.933049 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:01.933885 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:01.933890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:01.933892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:01.934015 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:01.934439 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:01.934442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:01.934445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:01.934709 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:01.934860 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:01.935596 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 00:50:01.935831 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:50:01.937369 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:01.918995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41913 (local address 127.0.222.1:38983)
0924 00:50:01.919769 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:01.919814 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:01.921740 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:01.922508 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:01.922517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:01.923133 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:01.923745 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:01.923764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:01.924795 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:01.924799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:01.926130 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:01.926143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:01.926947 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:01.926955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:01.927203 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:01.927851 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:01.927881 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:01.930333 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:01.933231 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:01.933241 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:01.933262 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:01.933743 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:01.934122 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:01.934127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:01.934131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:01.934319 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:01.934833 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:01.934841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:01.935209 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 00:50:01.935469 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 00:50:01.936025 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":318,"threads_started":1}
I20260924 00:50:01.939021 5256 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41913
I20260924 00:50:01.939369 5256 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260924 00:50:01.940105 5256 heartbeater.cc:511] Master 127.0.222.62:41913 requested a full tablet report, sending...
I20260924 00:50:01.942538 5067 ts_manager.cc:194] Registered new tserver with Master: fbb9facbbd484c0cad7e73487c25469e (127.0.222.1:37751)
I20260924 00:50:01.944535 5067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:38983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:50:01.965478 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41913
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:02.292719 5263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:02.293025 5263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:02.293123 5263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:02.301690 5263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:02.301815 5263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:02.301947 5263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:02.312194 5263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:02.313587 5263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:02.314891 5263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:02.327957 5269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:02.328116 5268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:02.328812 5271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:02.328917 5263 server_base.cc:1034] running on GCE node
I20260924 00:50:02.329773 5263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:02.330953 5263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:02.332198 5263 hybrid_clock.cc:648] HybridClock initialized: now 1790211002332145 us; error 37 us; skew 500 ppm
I20260924 00:50:02.332597 5263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:02 dist-test-slave-rtvz krb5kdc[5023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:02.336743 5263 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:50:02.338294 5263 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:02.338438 5263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:02.338800 5263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:02.342589 5263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "8a63931e257a486ba370dbc6a4b00034"
format_stamp: "Formatted at 2026-09-24 00:50:02 on dist-test-slave-rtvz"
server_key: "dc72e92b8ce15e674d6cd5cb10d3e7c1"
server_key_iv: "17e1735ab150b60117bb4e2c7bb4ea57"
server_key_version: "encryptionkey@0"
I20260924 00:50:02.343564 5263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "8a63931e257a486ba370dbc6a4b00034"
format_stamp: "Formatted at 2026-09-24 00:50:02 on dist-test-slave-rtvz"
server_key: "dc72e92b8ce15e674d6cd5cb10d3e7c1"
server_key_iv: "17e1735ab150b60117bb4e2c7bb4ea57"
server_key_version: "encryptionkey@0"
I20260924 00:50:02.348775 5263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 00:50:02.352447 5277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:02.353892 5263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:02.354048 5263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "8a63931e257a486ba370dbc6a4b00034"
format_stamp: "Formatted at 2026-09-24 00:50:02 on dist-test-slave-rtvz"
server_key: "dc72e92b8ce15e674d6cd5cb10d3e7c1"
server_key_iv: "17e1735ab150b60117bb4e2c7bb4ea57"
server_key_version: "encryptionkey@0"
I20260924 00:50:02.354275 5263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:02.374011 5263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:02.378290 5263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:02.378538 5263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:02.379518 5263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:02.380888 5263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:02.380980 5263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:02.381088 5263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:02.381151 5263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:02.412066 5263 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41887
I20260924 00:50:02.412122 5389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41887 every 8 connection(s)
I20260924 00:50:02.412925 5263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:02.414925 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 5263
I20260924 00:50:02.415084 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:50:02.415560 888 external_mini_cluster.cc:1428] Setting key f658c301a6cb744d6746ffe13af9cdeb
Sep 24 00:50:02 dist-test-slave-rtvz krb5kdc[5023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:50:02.434687 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.415582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:53299 (local address 127.0.222.62:41913)
0924 00:50:02.415781 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:02.415787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:02.417142 (+ 1355us) server_negotiation.cc:407] Connection header received
0924 00:50:02.419123 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:02.419129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:02.419210 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:02.419318 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:02.420900 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.421854 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.423473 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.424075 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.424199 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:02.427851 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:02.427859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:02.427863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:02.427908 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:02.431258 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:02.432091 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:02.432098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:02.432101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:02.432271 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:02.432756 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:02.432761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:02.432765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:02.433134 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:02.433380 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:02.434165 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 00:50:02.434383 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 00:50:02.435730 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.416107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41913 (local address 127.0.222.2:53299)
0924 00:50:02.416983 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:02.417029 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:02.418801 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:02.419511 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:02.419520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:02.420118 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:02.420707 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:02.420725 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.422014 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.422018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:02.423314 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:02.423327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.424298 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.424303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:02.424504 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:02.425146 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:02.425168 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:02.427569 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:02.431440 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:02.431448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:02.431464 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:02.431961 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:02.432424 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:02.432429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:02.432432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:02.432643 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:02.433358 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:02.433367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:02.433767 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 00:50:02.434043 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 00:50:02.434512 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":378,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:50:02.437441 5390 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41913
I20260924 00:50:02.437757 5390 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:02.438441 5390 heartbeater.cc:511] Master 127.0.222.62:41913 requested a full tablet report, sending...
I20260924 00:50:02.439919 5067 ts_manager.cc:194] Registered new tserver with Master: 8a63931e257a486ba370dbc6a4b00034 (127.0.222.2:41887)
I20260924 00:50:02.440699 5067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:53299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:50:02.461146 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41913
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:02.786180 5395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:02.786463 5395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:02.786597 5395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:02.794204 5395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:02.794334 5395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:02.794502 5395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:02.804441 5395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:02.806102 5395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:02.807531 5395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:02.821120 5400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:02.821408 5402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:02.821132 5404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:02.822093 5395 server_base.cc:1034] running on GCE node
I20260924 00:50:02.822889 5395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:02.824729 5395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:02.825997 5395 hybrid_clock.cc:648] HybridClock initialized: now 1790211002825914 us; error 64 us; skew 500 ppm
I20260924 00:50:02.826421 5395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:02 dist-test-slave-rtvz krb5kdc[5023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:02.830482 5395 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:50:02.831647 5395 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:02.831759 5395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:02.832087 5395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:02.835577 5395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "777ac385d210433ba9254d72e4207b85"
format_stamp: "Formatted at 2026-09-24 00:50:02 on dist-test-slave-rtvz"
server_key: "9e524690a534d5f16316ed31a0233fe9"
server_key_iv: "91bb6d47e2fde4226915e2d58cf53063"
server_key_version: "encryptionkey@0"
I20260924 00:50:02.836484 5395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "777ac385d210433ba9254d72e4207b85"
format_stamp: "Formatted at 2026-09-24 00:50:02 on dist-test-slave-rtvz"
server_key: "9e524690a534d5f16316ed31a0233fe9"
server_key_iv: "91bb6d47e2fde4226915e2d58cf53063"
server_key_version: "encryptionkey@0"
I20260924 00:50:02.842039 5395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 00:50:02.845620 5410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:02.847070 5395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:50:02.847236 5395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "777ac385d210433ba9254d72e4207b85"
format_stamp: "Formatted at 2026-09-24 00:50:02 on dist-test-slave-rtvz"
server_key: "9e524690a534d5f16316ed31a0233fe9"
server_key_iv: "91bb6d47e2fde4226915e2d58cf53063"
server_key_version: "encryptionkey@0"
I20260924 00:50:02.847477 5395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:02.868762 5395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:02.873322 5395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:02.873606 5395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:02.874612 5395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:02.876051 5395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:02.876152 5395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:02.876266 5395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:02.876344 5395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:02.907274 5395 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33451
I20260924 00:50:02.907334 5522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33451 every 8 connection(s)
I20260924 00:50:02.908208 5395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:02.910193 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 5395
I20260924 00:50:02.910386 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:50:02.910925 888 external_mini_cluster.cc:1428] Setting key b4786cba8f1effdb493cc71b8a0915c3
Sep 24 00:50:02 dist-test-slave-rtvz krb5kdc[5023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:02.930280 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.911751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:40115 (local address 127.0.222.62:41913)
0924 00:50:02.911962 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:02.911968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:02.912946 (+ 978us) server_negotiation.cc:407] Connection header received
0924 00:50:02.915069 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:02.915075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:02.915161 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:02.915243 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:02.916896 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.917813 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.919544 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.919986 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.920102 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:02.923671 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:02.923676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:02.923680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:02.923713 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:02.926635 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:02.927592 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:02.927598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:02.927601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:02.927760 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:02.928262 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:02.928267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:02.928270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:02.928603 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:02.928722 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:02.929724 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0924 00:50:02.929953 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:50:02.932515 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.911785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41913 (local address 127.0.222.3:40115)
0924 00:50:02.912688 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:02.912740 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:02.914799 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:02.915488 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:02.915496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:02.916058 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:02.916659 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:02.916678 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.918038 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.918043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:02.919364 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:02.919380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.920219 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.920226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:02.920452 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:02.921112 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:02.921135 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:02.923410 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:02.926809 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:02.926819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:02.926841 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:02.927476 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:02.927905 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:02.927910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:02.927914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:02.928141 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:02.928829 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:02.928837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:02.929307 (+ 470us) client_negotiation.cc:769] Sending connection context
0924 00:50:02.930577 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0924 00:50:02.931140 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":338,"threads_started":1}
I20260924 00:50:02.934566 5523 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41913
I20260924 00:50:02.934857 5523 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:02.935488 5523 heartbeater.cc:511] Master 127.0.222.62:41913 requested a full tablet report, sending...
I20260924 00:50:02.937013 5067 ts_manager.cc:194] Registered new tserver with Master: 777ac385d210433ba9254d72e4207b85 (127.0.222.3:33451)
I20260924 00:50:02.937870 5067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:40115
I20260924 00:50:02.940068 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:50:02.948524 5256 heartbeater.cc:503] Master 127.0.222.62:41913 was elected leader, sending a full tablet report...
I20260924 00:50:02.958092 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.944978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.0.222.62:41913)
0924 00:50:02.945142 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:02.945147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:02.945589 (+ 442us) server_negotiation.cc:407] Connection header received
0924 00:50:02.945718 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:02.945725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:02.945831 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:02.945936 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:02.947456 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.948201 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.949786 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.950319 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.950471 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:02.952743 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:02.952748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:02.952752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:02.952783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:02.955109 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:02.955800 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:02.955805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:02.955808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:02.955949 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:02.956323 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:02.956327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:02.956329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:02.956688 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:02.956921 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:02.957508 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 00:50:02.957754 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 00:50:02.962854 5067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:50:02.970511 5067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:03.004359 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.984670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.0.222.3:33451)
0924 00:50:02.993078 (+ 8408us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:02.993088 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:02.995134 (+ 2046us) server_negotiation.cc:407] Connection header received
0924 00:50:02.995229 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:02.995236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:02.995474 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:02.995667 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:02.996855 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.998565 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.001093 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.002572 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.002695 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.003002 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.003823 (+ 821us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.004002 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8162,"thread_start_us":7957,"threads_started":1}
I20260924 00:50:03.004380 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.983632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33451 (local address 127.0.0.1:34336)
0924 00:50:02.994303 (+ 10671us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:02.994372 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:02.994610 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:02.996167 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:02.996173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:02.996201 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:02.996635 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:02.996648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.998726 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.998731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.000928 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:03.000941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.003140 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.003147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.003353 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.003481 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 00:50:03.003648 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 00:50:03.003855 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10535,"mutex_wait_us":44,"spinlock_wait_cycles":30592,"thread_start_us":11151,"threads_started":1}
I20260924 00:50:03.004463 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.983360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41887 (local address 127.0.0.1:54968)
0924 00:50:02.994303 (+ 10943us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:02.994361 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:02.994610 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:02.996651 (+ 2041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:02.996655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:02.996678 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:02.997257 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:02.997271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.999690 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:02.999695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.002013 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:03.002028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.003612 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.003620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.003821 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.003853 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:50:03.003938 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:50:03.004067 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10608,"mutex_wait_us":53,"spinlock_wait_cycles":7168,"thread_start_us":14180,"threads_started":1}
I20260924 00:50:03.005163 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.983619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.0.222.2:41887)
0924 00:50:02.995997 (+ 12378us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:02.996007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:02.996040 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:50:02.996148 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:02.996157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:02.996423 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:02.996650 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:02.997582 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:02.999576 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.002201 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.003368 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.003510 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.003700 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.004712 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.004878 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12164,"thread_start_us":12290,"threads_started":1}
I20260924 00:50:03.006714 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.984007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:37751 (local address 127.0.0.1:55076)
0924 00:50:02.997874 (+ 13867us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:02.997897 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:02.998032 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:02.998961 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:02.998966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:02.998994 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:02.999461 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:02.999471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.001518 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.001523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.003760 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:03.003773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.005804 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.005811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.005989 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.006021 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:50:03.006111 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 00:50:03.006249 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13719,"mutex_wait_us":2,"thread_start_us":9760,"threads_started":1}
I20260924 00:50:03.007943 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:02.985324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.0.222.1:37751)
0924 00:50:02.992052 (+ 6728us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:02.992063 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:02.998016 (+ 5953us) server_negotiation.cc:407] Connection header received
0924 00:50:02.998169 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:02.998177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:02.998450 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:02.998646 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:02.999629 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.001359 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.003935 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.005273 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.005393 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.005655 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.007048 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.007592 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6497,"thread_start_us":6314,"threads_started":1}
I20260924 00:50:03.008770 5458 tablet_service.cc:1467] Processing CreateTablet for tablet 48ecc5e0941e49e38c72150be25ca3f2 (DEFAULT_TABLE table=test-table [id=2ca31a8cb60c4ba893099278eef4f977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:03.008915 5325 tablet_service.cc:1467] Processing CreateTablet for tablet 48ecc5e0941e49e38c72150be25ca3f2 (DEFAULT_TABLE table=test-table [id=2ca31a8cb60c4ba893099278eef4f977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:03.010239 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ecc5e0941e49e38c72150be25ca3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:03.010442 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ecc5e0941e49e38c72150be25ca3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:03.011945 5191 tablet_service.cc:1467] Processing CreateTablet for tablet 48ecc5e0941e49e38c72150be25ca3f2 (DEFAULT_TABLE table=test-table [id=2ca31a8cb60c4ba893099278eef4f977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:03.013537 5191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ecc5e0941e49e38c72150be25ca3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:03.023326 5541 tablet_bootstrap.cc:492] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Bootstrap starting.
I20260924 00:50:03.027475 5541 tablet_bootstrap.cc:654] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:03.028520 5541 log.cc:824] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:03.029657 5543 tablet_bootstrap.cc:492] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Bootstrap starting.
I20260924 00:50:03.030958 5542 tablet_bootstrap.cc:492] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Bootstrap starting.
I20260924 00:50:03.033413 5543 tablet_bootstrap.cc:654] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:03.034003 5541 tablet_bootstrap.cc:492] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: No bootstrap required, opened a new log
I20260924 00:50:03.034322 5541 ts_tablet_manager.cc:1397] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260924 00:50:03.034860 5543 log.cc:824] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:03.035254 5542 tablet_bootstrap.cc:654] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:03.036711 5542 log.cc:824] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:03.038794 5543 tablet_bootstrap.cc:492] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: No bootstrap required, opened a new log
I20260924 00:50:03.039083 5543 ts_tablet_manager.cc:1397] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 00:50:03.039961 5542 tablet_bootstrap.cc:492] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: No bootstrap required, opened a new log
I20260924 00:50:03.040258 5542 ts_tablet_manager.cc:1397] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 00:50:03.045286 5541 raft_consensus.cc:348] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.045888 5541 raft_consensus.cc:374] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:03.046016 5541 raft_consensus.cc:729] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a63931e257a486ba370dbc6a4b00034, State: Initialized, Role: FOLLOWER
I20260924 00:50:03.046891 5541 consensus_queue.cc:260] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.048382 5390 heartbeater.cc:503] Master 127.0.222.62:41913 was elected leader, sending a full tablet report...
I20260924 00:50:03.048211 5543 raft_consensus.cc:348] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.048731 5543 raft_consensus.cc:374] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:03.048838 5543 raft_consensus.cc:729] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb9facbbd484c0cad7e73487c25469e, State: Initialized, Role: FOLLOWER
I20260924 00:50:03.049053 5541 ts_tablet_manager.cc:1428] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260924 00:50:03.049655 5543 consensus_queue.cc:260] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.051295 5542 raft_consensus.cc:348] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.051580 5543 ts_tablet_manager.cc:1428] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 00:50:03.051656 5542 raft_consensus.cc:374] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:03.051756 5542 raft_consensus.cc:729] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777ac385d210433ba9254d72e4207b85, State: Initialized, Role: FOLLOWER
I20260924 00:50:03.052500 5542 consensus_queue.cc:260] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.053821 5523 heartbeater.cc:503] Master 127.0.222.62:41913 was elected leader, sending a full tablet report...
I20260924 00:50:03.054441 5542 ts_tablet_manager.cc:1428] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260924 00:50:03.159821 5524 tablet.cc:2369] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:50:03.164870 5391 tablet.cc:2369] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:50:03.168040 5257 tablet.cc:2369] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:50:03.439486 5547 raft_consensus.cc:482] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:03.439827 5547 raft_consensus.cc:504] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.441672 5547 leader_election.cc:288] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 777ac385d210433ba9254d72e4207b85 (127.0.222.3:33451), fbb9facbbd484c0cad7e73487c25469e (127.0.222.1:37751)
I20260924 00:50:03.448916 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:03.442121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:47357 (local address 127.0.222.1:37751)
0924 00:50:03.442331 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:03.442336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:03.443478 (+ 1142us) server_negotiation.cc:407] Connection header received
0924 00:50:03.443651 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:03.443656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:03.443718 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:03.443789 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:03.444480 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.445261 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.447128 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.447961 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.448082 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.448187 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.448538 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.448658 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:50:03.448915 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:03.442104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:56853 (local address 127.0.222.3:33451)
0924 00:50:03.442304 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:03.442310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:03.443350 (+ 1040us) server_negotiation.cc:407] Connection header received
0924 00:50:03.443565 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:03.443569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:03.443629 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:03.443702 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:03.444337 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.445116 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.447107 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.447951 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.448065 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.448185 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.448520 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.448644 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:50:03.449404 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:03.442159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33451 (local address 127.0.222.2:56853)
0924 00:50:03.443240 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:03.443256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:03.443419 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:03.443729 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:03.443734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:03.443757 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:03.444196 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:03.444205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.445274 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.445280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.446954 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:03.446967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.448197 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.448204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.448366 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.448399 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:50:03.448482 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:50:03.448615 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":37,"thread_start_us":565,"threads_started":1}
I20260924 00:50:03.449446 5551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:03.442189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:37751 (local address 127.0.222.2:47357)
0924 00:50:03.443340 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:03.443354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:03.443518 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:03.443813 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:03.443822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:03.443888 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:03.444331 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:03.444340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.445386 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.445390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.446983 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:03.446995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.448247 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.448252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:03.448399 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.448422 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 00:50:03.448498 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:50:03.448620 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":16,"spinlock_wait_cycles":2816,"thread_start_us":741,"threads_started":1}
I20260924 00:50:03.450460 5478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ecc5e0941e49e38c72150be25ca3f2" candidate_uuid: "8a63931e257a486ba370dbc6a4b00034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777ac385d210433ba9254d72e4207b85" is_pre_election: true
I20260924 00:50:03.450518 5211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ecc5e0941e49e38c72150be25ca3f2" candidate_uuid: "8a63931e257a486ba370dbc6a4b00034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb9facbbd484c0cad7e73487c25469e" is_pre_election: true
I20260924 00:50:03.451025 5478 raft_consensus.cc:2454] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a63931e257a486ba370dbc6a4b00034 in term 0.
I20260924 00:50:03.451151 5211 raft_consensus.cc:2454] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a63931e257a486ba370dbc6a4b00034 in term 0.
I20260924 00:50:03.451717 5281 leader_election.cc:302] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 777ac385d210433ba9254d72e4207b85, 8a63931e257a486ba370dbc6a4b00034; no voters:
I20260924 00:50:03.452116 5547 raft_consensus.cc:2781] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:03.452258 5547 raft_consensus.cc:482] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:03.452360 5547 raft_consensus.cc:3037] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:03.454813 5547 raft_consensus.cc:504] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.455459 5547 leader_election.cc:288] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [CANDIDATE]: Term 1 election: Requested vote from peers 777ac385d210433ba9254d72e4207b85 (127.0.222.3:33451), fbb9facbbd484c0cad7e73487c25469e (127.0.222.1:37751)
I20260924 00:50:03.456084 5478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ecc5e0941e49e38c72150be25ca3f2" candidate_uuid: "8a63931e257a486ba370dbc6a4b00034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777ac385d210433ba9254d72e4207b85"
I20260924 00:50:03.456123 5211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ecc5e0941e49e38c72150be25ca3f2" candidate_uuid: "8a63931e257a486ba370dbc6a4b00034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb9facbbd484c0cad7e73487c25469e"
I20260924 00:50:03.456300 5478 raft_consensus.cc:3037] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:03.456346 5211 raft_consensus.cc:3037] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:03.459048 5478 raft_consensus.cc:2454] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a63931e257a486ba370dbc6a4b00034 in term 1.
I20260924 00:50:03.459334 5211 raft_consensus.cc:2454] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a63931e257a486ba370dbc6a4b00034 in term 1.
I20260924 00:50:03.459602 5281 leader_election.cc:302] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 777ac385d210433ba9254d72e4207b85, 8a63931e257a486ba370dbc6a4b00034; no voters:
I20260924 00:50:03.460007 5547 raft_consensus.cc:2781] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:03.460456 5547 raft_consensus.cc:686] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 1 LEADER]: Becoming Leader. State: Replica: 8a63931e257a486ba370dbc6a4b00034, State: Running, Role: LEADER
I20260924 00:50:03.461006 5547 consensus_queue.cc:237] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } }
I20260924 00:50:03.465920 5067 catalog_manager.cc:5495] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a63931e257a486ba370dbc6a4b00034 (127.0.222.2). New cstate: current_term: 1 leader_uuid: "8a63931e257a486ba370dbc6a4b00034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a63931e257a486ba370dbc6a4b00034" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 } health_report { overall_health: UNKNOWN } } }
I20260924 00:50:03.575826 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:03.569827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.0.222.2:41887)
0924 00:50:03.570503 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:03.570507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:03.570524 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:50:03.570612 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:03.570616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:03.570697 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:03.570831 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:03.571249 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.571903 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.573382 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.573806 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.573906 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.574439 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:03.575095 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:03.575202 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.575380 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.575461 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":457,"threads_started":1}
I20260924 00:50:03.584143 5478 raft_consensus.cc:1261] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 1 FOLLOWER]: Refusing update from remote peer 8a63931e257a486ba370dbc6a4b00034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:03.584167 5211 raft_consensus.cc:1261] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 1 FOLLOWER]: Refusing update from remote peer 8a63931e257a486ba370dbc6a4b00034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:03.585119 5547 consensus_queue.cc:1035] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb9facbbd484c0cad7e73487c25469e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:03.585518 5552 consensus_queue.cc:1035] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "777ac385d210433ba9254d72e4207b85" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:03.597189 5557 mvcc.cc:205] Tried to move back new op lower bound from 7332704270670475264 to 7332704270180249600. Current Snapshot: MvccSnapshot[applied={T|T < 7332704270670475264}]
I20260924 00:50:03.611119 5067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:03.611399 5067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:03.615348 5067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28d0947daaad44afbbbd1060a3f50fbc: Sending DeleteTablet for 3 replicas of tablet 48ecc5e0941e49e38c72150be25ca3f2
I20260924 00:50:03.616784 5458 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ecc5e0941e49e38c72150be25ca3f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:34336
I20260924 00:50:03.616797 5325 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ecc5e0941e49e38c72150be25ca3f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:54968
I20260924 00:50:03.617127 5191 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ecc5e0941e49e38c72150be25ca3f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:55076
I20260924 00:50:03.619184 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 5129
I20260924 00:50:03.629294 5571 tablet_replica.cc:330] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: stopping tablet replica
I20260924 00:50:03.629962 5571 raft_consensus.cc:2229] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:03.630604 5571 raft_consensus.cc:2258] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:03.632752 5571 ts_tablet_manager.cc:1905] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:03.633541 5572 tablet_replica.cc:330] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: stopping tablet replica
I20260924 00:50:03.634011 5572 raft_consensus.cc:2229] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:03.634577 5572 raft_consensus.cc:2258] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:03.636595 5572 ts_tablet_manager.cc:1905] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:03.642254 5571 ts_tablet_manager.cc:1918] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:03.642730 5571 log.cc:1196] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/48ecc5e0941e49e38c72150be25ca3f2
I20260924 00:50:03.643405 5571 ts_tablet_manager.cc:1939] T 48ecc5e0941e49e38c72150be25ca3f2 P 777ac385d210433ba9254d72e4207b85: Deleting consensus metadata
I20260924 00:50:03.644331 5572 ts_tablet_manager.cc:1918] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:03.644618 5572 log.cc:1196] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/48ecc5e0941e49e38c72150be25ca3f2
I20260924 00:50:03.645218 5572 ts_tablet_manager.cc:1939] T 48ecc5e0941e49e38c72150be25ca3f2 P fbb9facbbd484c0cad7e73487c25469e: Deleting consensus metadata
I20260924 00:50:03.645918 5054 catalog_manager.cc:4841] TS 777ac385d210433ba9254d72e4207b85 (127.0.222.3:33451): tablet 48ecc5e0941e49e38c72150be25ca3f2 (table test-table [id=2ca31a8cb60c4ba893099278eef4f977]) successfully deleted
I20260924 00:50:03.646842 5570 tablet_replica.cc:330] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: stopping tablet replica
I20260924 00:50:03.647403 5570 raft_consensus.cc:2229] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:50:03.647512 5051 catalog_manager.cc:4841] TS fbb9facbbd484c0cad7e73487c25469e (127.0.222.1:37751): tablet 48ecc5e0941e49e38c72150be25ca3f2 (table test-table [id=2ca31a8cb60c4ba893099278eef4f977]) successfully deleted
I20260924 00:50:03.648361 5570 raft_consensus.cc:2258] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:03.650424 5570 ts_tablet_manager.cc:1905] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 00:50:03 dist-test-slave-rtvz krb5kdc[5023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:50:03.658924 5570 ts_tablet_manager.cc:1918] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:03.659138 5570 log.cc:1196] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/48ecc5e0941e49e38c72150be25ca3f2
I20260924 00:50:03.659823 5570 ts_tablet_manager.cc:1939] T 48ecc5e0941e49e38c72150be25ca3f2 P 8a63931e257a486ba370dbc6a4b00034: Deleting consensus metadata
I20260924 00:50:03.660578 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:03.620883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.0.222.1:37751)
0924 00:50:03.621078 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:03.621086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:03.644044 (+ 22958us) server_negotiation.cc:407] Connection header received
0924 00:50:03.644595 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:03.644605 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:03.644713 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:03.644796 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:03.646609 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.647727 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.649295 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.649852 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.649967 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.654197 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:03.654211 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:03.654227 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:03.654267 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:03.657143 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:03.658043 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:03.658051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:03.658054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:03.658205 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:03.658687 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:03.658692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:03.658694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:03.659109 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:03.659435 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.659833 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.660074 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:50:03.661331 5251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:03.661866 5053 catalog_manager.cc:4841] TS 8a63931e257a486ba370dbc6a4b00034 (127.0.222.2:41887): tablet 48ecc5e0941e49e38c72150be25ca3f2 (table test-table [id=2ca31a8cb60c4ba893099278eef4f977]) successfully deleted
I20260924 00:50:03.750478 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 5129
I20260924 00:50:03.771009 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 5263
Sep 24 00:50:03 dist-test-slave-rtvz krb5kdc[5023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:50:03.789368 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:03.771966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54978 (local address 127.0.222.2:41887)
0924 00:50:03.772120 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:03.772124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:03.772141 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:50:03.772194 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:03.772201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:03.772261 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:03.772340 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:03.774045 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.774986 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.776917 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.777433 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.777550 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.782366 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:03.782384 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:03.782404 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:03.782451 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:03.786464 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:03.787197 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:03.787206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:03.787210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:03.787363 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:03.787674 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:03.787679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:03.787682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:03.788143 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:03.788458 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.788684 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.788946 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 00:50:03.790170 5385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:03.902953 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 5263
I20260924 00:50:03.917569 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 5395
Sep 24 00:50:03 dist-test-slave-rtvz krb5kdc[5023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:50:03.932578 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:03.918296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.0.222.3:33451)
0924 00:50:03.918471 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:03.918477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:03.918558 (+ 81us) server_negotiation.cc:407] Connection header received
0924 00:50:03.918735 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:03.918741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:03.918922 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:03.919007 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:03.920267 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.921071 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.922321 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:03.922748 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:03.922862 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:03.927045 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:03.927058 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:03.927078 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:03.927134 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:03.929860 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:03.930543 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:03.930550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:03.930553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:03.930697 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:03.931066 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:03.931070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:03.931072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:03.931445 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:03.931706 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:03.932062 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 00:50:03.932273 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 00:50:03.933285 5518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:04.021490 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 5395
I20260924 00:50:04.034243 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 5036
I20260924 00:50:04.035444 5097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:04.143989 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 5036
2026-09-24T00:50:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4521 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T00:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:04Z Disabled control of system clock
I20260924 00:50:04.206513 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42509
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:50:04.536172 5581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:04.536501 5581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:04.536588 5581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:04.544061 5581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:04.544183 5581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:04.544219 5581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:04.544270 5581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:04.544312 5581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:04.553794 5581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:04.555119 5581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:04.556943 5581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:04.569049 5586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:04.569010 5587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:04.569024 5589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:04.569700 5581 server_base.cc:1034] running on GCE node
I20260924 00:50:04.570568 5581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:04.572084 5581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:04.573300 5581 hybrid_clock.cc:648] HybridClock initialized: now 1790211004573247 us; error 48 us; skew 500 ppm
I20260924 00:50:04.573662 5581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:04.576030 5581 webserver.cc:460] Webserver started at http://127.0.222.62:38281/ using document root <none> and password file <none>
I20260924 00:50:04.576754 5581 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:04.576862 5581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:04.577226 5581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:04.580271 5581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "ee8f5a0f1668412ba5cec6d468855f1a"
format_stamp: "Formatted at 2026-09-24 00:50:04 on dist-test-slave-rtvz"
server_key: "6e43a3a98b0baf1a5335886db22ef463"
server_key_iv: "6635d48277e5e96b18d8407374f7de5b"
server_key_version: "encryptionkey@0"
I20260924 00:50:04.581126 5581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "ee8f5a0f1668412ba5cec6d468855f1a"
format_stamp: "Formatted at 2026-09-24 00:50:04 on dist-test-slave-rtvz"
server_key: "6e43a3a98b0baf1a5335886db22ef463"
server_key_iv: "6635d48277e5e96b18d8407374f7de5b"
server_key_version: "encryptionkey@0"
I20260924 00:50:04.586771 5581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 00:50:04.590587 5595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:04.592270 5581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:50:04.592479 5581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "ee8f5a0f1668412ba5cec6d468855f1a"
format_stamp: "Formatted at 2026-09-24 00:50:04 on dist-test-slave-rtvz"
server_key: "6e43a3a98b0baf1a5335886db22ef463"
server_key_iv: "6635d48277e5e96b18d8407374f7de5b"
server_key_version: "encryptionkey@0"
I20260924 00:50:04.592712 5581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:04.599152 5581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:04.600037 5581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:04.600333 5581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:04.619168 5581 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42509
I20260924 00:50:04.619191 5647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42509 every 8 connection(s)
I20260924 00:50:04.620656 5581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:04.624593 5648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:04.625452 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 5581
I20260924 00:50:04.625638 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:04.626340 888 external_mini_cluster.cc:1428] Setting key 44698983a1218530791fa2479804de49
I20260924 00:50:04.635082 5651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:04.628427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.0.222.62:42509)
0924 00:50:04.629661 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:04.629674 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:04.629721 (+ 47us) server_negotiation.cc:407] Connection header received
0924 00:50:04.631270 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:04.631295 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:04.631777 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:04.632169 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:04.633357 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:04.633363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:04.633377 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:04.633392 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:04.633469 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:04.633488 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:04.633692 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:04.633828 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 00:50:04.634102 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":483,"threads_started":1}
I20260924 00:50:04.638294 5648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a: Bootstrap starting.
I20260924 00:50:04.641363 5648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:04.642395 5648 log.cc:824] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:04.645491 5648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a: No bootstrap required, opened a new log
I20260924 00:50:04.652668 5648 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8f5a0f1668412ba5cec6d468855f1a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42509 } }
I20260924 00:50:04.653051 5648 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:04.653141 5648 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8f5a0f1668412ba5cec6d468855f1a, State: Initialized, Role: FOLLOWER
I20260924 00:50:04.653771 5648 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8f5a0f1668412ba5cec6d468855f1a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42509 } }
I20260924 00:50:04.653982 5648 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:04.654081 5648 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:04.654204 5648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:04.656579 5648 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8f5a0f1668412ba5cec6d468855f1a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42509 } }
I20260924 00:50:04.657086 5648 leader_election.cc:302] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee8f5a0f1668412ba5cec6d468855f1a; no voters:
I20260924 00:50:04.657728 5648 leader_election.cc:288] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:04.657861 5653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:04.658533 5653 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [term 1 LEADER]: Becoming Leader. State: Replica: ee8f5a0f1668412ba5cec6d468855f1a, State: Running, Role: LEADER
I20260924 00:50:04.659106 5653 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8f5a0f1668412ba5cec6d468855f1a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42509 } }
I20260924 00:50:04.659842 5648 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:04.661895 5655 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee8f5a0f1668412ba5cec6d468855f1a. Latest consensus state: current_term: 1 leader_uuid: "ee8f5a0f1668412ba5cec6d468855f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8f5a0f1668412ba5cec6d468855f1a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42509 } } }
I20260924 00:50:04.661928 5654 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee8f5a0f1668412ba5cec6d468855f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee8f5a0f1668412ba5cec6d468855f1a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42509 } } }
I20260924 00:50:04.662293 5655 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:04.662288 5654 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:04.665040 5660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:04.670271 5660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:04.680138 5660 catalog_manager.cc:1347] Generated new cluster ID: aa84241bcb57499588ca949514f32adf
I20260924 00:50:04.680296 5660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:04.699096 5660 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:04.699604 5660 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:04.716759 5660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a: Generated new TSK 0
I20260924 00:50:04.717761 5660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:50:04.724354 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42509
--builtin_ntp_servers=127.0.222.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:50:05.087977 5672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:05.088268 5672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:05.088358 5672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:05.095918 5672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:05.096040 5672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:05.096158 5672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:05.105726 5672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:05.106984 5672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:05.108808 5672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:05.122431 5678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:05.122481 5680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:05.122527 5677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:05.123505 5672 server_base.cc:1034] running on GCE node
I20260924 00:50:05.124251 5672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:05.125281 5672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:05.126503 5672 hybrid_clock.cc:648] HybridClock initialized: now 1790211005126449 us; error 45 us; skew 500 ppm
I20260924 00:50:05.126906 5672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:05.129371 5672 webserver.cc:460] Webserver started at http://127.0.222.1:32849/ using document root <none> and password file <none>
I20260924 00:50:05.130146 5672 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:05.130254 5672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:05.130558 5672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:05.133551 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "6e890d986ad44e24a8167874b179a1d4"
format_stamp: "Formatted at 2026-09-24 00:50:05 on dist-test-slave-rtvz"
server_key: "ccaa7b6d4f5ff8c8a583b60e002f601e"
server_key_iv: "ce5b94f3d8da9e140bd555f553e45866"
server_key_version: "encryptionkey@0"
I20260924 00:50:05.134467 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "6e890d986ad44e24a8167874b179a1d4"
format_stamp: "Formatted at 2026-09-24 00:50:05 on dist-test-slave-rtvz"
server_key: "ccaa7b6d4f5ff8c8a583b60e002f601e"
server_key_iv: "ce5b94f3d8da9e140bd555f553e45866"
server_key_version: "encryptionkey@0"
I20260924 00:50:05.139823 5672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:50:05.143621 5686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:05.145190 5672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:05.145423 5672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "6e890d986ad44e24a8167874b179a1d4"
format_stamp: "Formatted at 2026-09-24 00:50:05 on dist-test-slave-rtvz"
server_key: "ccaa7b6d4f5ff8c8a583b60e002f601e"
server_key_iv: "ce5b94f3d8da9e140bd555f553e45866"
server_key_version: "encryptionkey@0"
I20260924 00:50:05.145653 5672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:05.151561 5672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:05.152411 5672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:05.152654 5672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:05.153618 5672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:05.155004 5672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:05.155087 5672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:05.155190 5672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:05.155263 5672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:05.189105 5672 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:36599
I20260924 00:50:05.189160 5799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:36599 every 8 connection(s)
I20260924 00:50:05.190547 5672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:05.198767 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 5672
I20260924 00:50:05.199015 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:50:05.199112 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:05.193266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:39847 (local address 127.0.222.62:42509)
0924 00:50:05.193706 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:05.193710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:05.194986 (+ 1276us) server_negotiation.cc:407] Connection header received
0924 00:50:05.196745 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:05.196751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:05.196825 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:05.196943 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:05.198346 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:05.198352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:05.198356 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:05.198374 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:05.198421 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:05.198433 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:05.198549 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:05.198732 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 00:50:05.198804 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":217,"threads_started":1}
I20260924 00:50:05.199678 888 external_mini_cluster.cc:1428] Setting key e68051476575d2e28fa99c242a054a34
I20260924 00:50:05.200102 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:05.193854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42509 (local address 127.0.222.1:39847)
0924 00:50:05.194767 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:05.194811 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:05.196467 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:05.197192 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:05.197234 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:05.197352 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:05.198039 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:05.198041 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:05.198081 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:05.198172 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:05.198521 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:05.198531 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:05.198539 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 00:50:05.198803 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 00:50:05.199094 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":252,"threads_started":1}
I20260924 00:50:05.201701 5800 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42509
I20260924 00:50:05.201980 5800 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:05.202562 5800 heartbeater.cc:511] Master 127.0.222.62:42509 requested a full tablet report, sending...
I20260924 00:50:05.202608 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42509
--builtin_ntp_servers=127.0.222.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 00:50:05.204676 5612 ts_manager.cc:194] Registered new tserver with Master: 6e890d986ad44e24a8167874b179a1d4 (127.0.222.1:36599)
W20260924 00:50:05.535202 5804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:05.535537 5804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:05.535669 5804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:05.543128 5804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:05.543255 5804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:05.543412 5804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:05.552873 5804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:05.554236 5804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:05.556030 5804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:05.569761 5810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:05.569908 5809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:05.569983 5812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:05.571120 5804 server_base.cc:1034] running on GCE node
I20260924 00:50:05.571852 5804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:05.572871 5804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:05.574126 5804 hybrid_clock.cc:648] HybridClock initialized: now 1790211005574057 us; error 48 us; skew 500 ppm
I20260924 00:50:05.574468 5804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:05.576979 5804 webserver.cc:460] Webserver started at http://127.0.222.2:34763/ using document root <none> and password file <none>
I20260924 00:50:05.577718 5804 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:05.577827 5804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:05.578142 5804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:05.581375 5804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "8f614b9268304dc19af77f4a397b9689"
format_stamp: "Formatted at 2026-09-24 00:50:05 on dist-test-slave-rtvz"
server_key: "7a7e4bde2b1495233888b965ff75aa9d"
server_key_iv: "40d2dbfb81e34f91ac5e477a9636bee5"
server_key_version: "encryptionkey@0"
I20260924 00:50:05.582230 5804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "8f614b9268304dc19af77f4a397b9689"
format_stamp: "Formatted at 2026-09-24 00:50:05 on dist-test-slave-rtvz"
server_key: "7a7e4bde2b1495233888b965ff75aa9d"
server_key_iv: "40d2dbfb81e34f91ac5e477a9636bee5"
server_key_version: "encryptionkey@0"
I20260924 00:50:05.587633 5804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:50:05.591451 5818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:05.592962 5804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:05.593132 5804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "8f614b9268304dc19af77f4a397b9689"
format_stamp: "Formatted at 2026-09-24 00:50:05 on dist-test-slave-rtvz"
server_key: "7a7e4bde2b1495233888b965ff75aa9d"
server_key_iv: "40d2dbfb81e34f91ac5e477a9636bee5"
server_key_version: "encryptionkey@0"
I20260924 00:50:05.593405 5804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:05.599292 5804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:05.600152 5804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:05.600416 5804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:05.601440 5804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:05.602869 5804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:05.602975 5804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:05.603082 5804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:05.603152 5804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:05.636348 5804 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:33905
I20260924 00:50:05.636420 5931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:33905 every 8 connection(s)
I20260924 00:50:05.637660 5804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:05.641808 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 5804
I20260924 00:50:05.641973 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:50:05.642374 888 external_mini_cluster.cc:1428] Setting key 505461f4013ebf0912a2934fd55f80b7
I20260924 00:50:05.645179 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42509
--builtin_ntp_servers=127.0.222.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 00:50:05.646064 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:05.640103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:36403 (local address 127.0.222.62:42509)
0924 00:50:05.640259 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:05.640263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:05.641709 (+ 1446us) server_negotiation.cc:407] Connection header received
0924 00:50:05.643591 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:05.643598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:05.643666 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:05.643752 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:05.645244 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:05.645251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:05.645256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:05.645277 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:05.645315 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:05.645331 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:05.645459 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:05.645709 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 00:50:05.645782 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:50:05.647089 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:05.640563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42509 (local address 127.0.222.2:36403)
0924 00:50:05.641538 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:05.641577 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:05.643310 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:05.644012 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:05.644024 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:05.644159 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:05.644845 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:05.644848 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:05.644945 (+ 97us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:05.645040 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:05.645507 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:05.645515 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:05.645521 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:50:05.645779 (+ 258us) client_negotiation.cc:240] Negotiation successful
0924 00:50:05.646072 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":293,"threads_started":1}
I20260924 00:50:05.648610 5932 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42509
I20260924 00:50:05.649012 5932 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:05.649796 5932 heartbeater.cc:511] Master 127.0.222.62:42509 requested a full tablet report, sending...
I20260924 00:50:05.651342 5612 ts_manager.cc:194] Registered new tserver with Master: 8f614b9268304dc19af77f4a397b9689 (127.0.222.2:33905)
W20260924 00:50:05.973434 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:05.973794 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:05.973938 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:05.981362 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:05.981492 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:05.981634 5935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:05.990962 5935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:05.992269 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:05.994340 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:06.008129 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:06.008224 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:06.008991 5935 server_base.cc:1034] running on GCE node
W20260924 00:50:06.008981 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:06.009824 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:06.010944 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:06.012154 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790211006012096 us; error 63 us; skew 500 ppm
I20260924 00:50:06.012502 5935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:06.015024 5935 webserver.cc:460] Webserver started at http://127.0.222.3:45563/ using document root <none> and password file <none>
I20260924 00:50:06.015756 5935 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:06.015862 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:06.016175 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:06.019224 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "6237c77aac96458b8cb3bdbd4923d5e1"
format_stamp: "Formatted at 2026-09-24 00:50:06 on dist-test-slave-rtvz"
server_key: "ee9aab76387c1b8ed17edd691982bc55"
server_key_iv: "fb726d858115eab531c4051997f807b5"
server_key_version: "encryptionkey@0"
I20260924 00:50:06.020049 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "6237c77aac96458b8cb3bdbd4923d5e1"
format_stamp: "Formatted at 2026-09-24 00:50:06 on dist-test-slave-rtvz"
server_key: "ee9aab76387c1b8ed17edd691982bc55"
server_key_iv: "fb726d858115eab531c4051997f807b5"
server_key_version: "encryptionkey@0"
I20260924 00:50:06.025790 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 00:50:06.029606 5949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:06.031028 5935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:50:06.031203 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "6237c77aac96458b8cb3bdbd4923d5e1"
format_stamp: "Formatted at 2026-09-24 00:50:06 on dist-test-slave-rtvz"
server_key: "ee9aab76387c1b8ed17edd691982bc55"
server_key_iv: "fb726d858115eab531c4051997f807b5"
server_key_version: "encryptionkey@0"
I20260924 00:50:06.031471 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:06.037631 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:06.038578 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:06.038854 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:06.039883 5935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:06.041556 5935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:06.041659 5935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:06.041771 5935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:06.041841 5935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:06.073719 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:45985
I20260924 00:50:06.073781 6062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:45985 every 8 connection(s)
I20260924 00:50:06.075095 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:06.083600 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.077742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:57991 (local address 127.0.222.62:42509)
0924 00:50:06.077965 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.077970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.079601 (+ 1631us) server_negotiation.cc:407] Connection header received
0924 00:50:06.081333 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.081339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.081406 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.081525 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.082902 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.082907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.082911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.082932 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.082970 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.082984 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.083106 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.083275 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.083351 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 00:50:06.084563 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.078365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42509 (local address 127.0.222.3:57991)
0924 00:50:06.079414 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.079460 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:06.081013 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:06.081757 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:06.081766 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:06.081876 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:06.082538 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:06.082540 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:06.082584 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:06.082700 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:06.083077 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:06.083083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:06.083089 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:50:06.083305 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 00:50:06.083550 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":291,"threads_started":1}
I20260924 00:50:06.084761 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 5935
I20260924 00:50:06.084980 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:50:06.085601 888 external_mini_cluster.cc:1428] Setting key c4b0815c125631a4fb54f74333a8967f
I20260924 00:50:06.086304 6063 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42509
I20260924 00:50:06.086644 6063 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:06.087289 6063 heartbeater.cc:511] Master 127.0.222.62:42509 requested a full tablet report, sending...
I20260924 00:50:06.088766 5612 ts_manager.cc:194] Registered new tserver with Master: 6237c77aac96458b8cb3bdbd4923d5e1 (127.0.222.3:45985)
I20260924 00:50:06.101496 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:50:06.108381 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.106084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.0.222.62:42509)
0924 00:50:06.106288 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.106292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.106778 (+ 486us) server_negotiation.cc:407] Connection header received
0924 00:50:06.106918 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.106922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.106978 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.107050 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.107891 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.107895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.107899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.107916 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.107947 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.107960 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.108056 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.108126 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.108173 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 00:50:06.112555 5612 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:50:06.119587 5612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:06.136868 6075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.131611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:45985 (local address 127.0.0.1:45866)
0924 00:50:06.132538 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.132595 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:06.132741 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:06.133864 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:06.133872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:06.133964 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:06.135558 (+ 1594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:06.135561 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:06.135592 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:06.135680 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:06.136110 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:06.136120 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:06.136128 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 00:50:06.136309 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 00:50:06.136403 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":397,"threads_started":1}
I20260924 00:50:06.137010 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.131782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.0.222.3:45985)
0924 00:50:06.133349 (+ 1567us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.133355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.133383 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:50:06.133453 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.133459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.133659 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.133814 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.135881 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.135891 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.135907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.135925 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.135989 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.136015 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.136183 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.136546 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.136647 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1387,"thread_start_us":1227,"threads_started":1}
I20260924 00:50:06.137392 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.132851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33905 (local address 127.0.0.1:43988)
0924 00:50:06.133334 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.133356 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:06.133466 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:06.134920 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:06.134925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:06.134950 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:06.135558 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:06.135561 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:06.135592 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:06.135680 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:06.136612 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:06.136617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:06.136626 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:50:06.136751 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 00:50:06.136822 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":755,"threads_started":1}
I20260924 00:50:06.138056 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.133456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.0.222.2:33905)
0924 00:50:06.133991 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.133998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.134024 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:50:06.134088 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.134095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.134307 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.134449 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.136279 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.136287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.136299 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.136325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.136381 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.136488 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.137561 (+ 1073us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.137629 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.137726 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":507,"threads_started":1}
I20260924 00:50:06.139200 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.134734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.0.222.1:36599)
0924 00:50:06.135959 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.135968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.136200 (+ 232us) server_negotiation.cc:407] Connection header received
0924 00:50:06.136276 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.136283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.136514 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.136644 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.138520 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.138528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.138539 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.138554 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.138601 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.138615 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.138734 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.138824 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.138933 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":1430,"threads_started":1}
I20260924 00:50:06.139727 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.133869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.0.1:41584)
0924 00:50:06.135974 (+ 2105us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.135995 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:06.136087 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:06.137475 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:06.137479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:06.137502 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:06.138299 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:06.138302 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:06.138318 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:06.138375 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:06.138705 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:06.138709 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:06.138713 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:50:06.138839 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 00:50:06.138888 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1992,"thread_start_us":1614,"threads_started":1}
I20260924 00:50:06.141520 5997 tablet_service.cc:1467] Processing CreateTablet for tablet 594b5d456fd846bf9adb003b2a93aa24 (DEFAULT_TABLE table=test-table [id=dcffaf72c28840be86c84d70db2b6748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:06.141516 5866 tablet_service.cc:1467] Processing CreateTablet for tablet 594b5d456fd846bf9adb003b2a93aa24 (DEFAULT_TABLE table=test-table [id=dcffaf72c28840be86c84d70db2b6748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:06.143044 5866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594b5d456fd846bf9adb003b2a93aa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:06.143040 5997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594b5d456fd846bf9adb003b2a93aa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:06.143271 5735 tablet_service.cc:1467] Processing CreateTablet for tablet 594b5d456fd846bf9adb003b2a93aa24 (DEFAULT_TABLE table=test-table [id=dcffaf72c28840be86c84d70db2b6748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:06.144668 5735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594b5d456fd846bf9adb003b2a93aa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:06.156837 6081 tablet_bootstrap.cc:492] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: Bootstrap starting.
I20260924 00:50:06.157920 6083 tablet_bootstrap.cc:492] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Bootstrap starting.
I20260924 00:50:06.159500 6082 tablet_bootstrap.cc:492] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Bootstrap starting.
I20260924 00:50:06.161832 6083 tablet_bootstrap.cc:654] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:06.161824 6081 tablet_bootstrap.cc:654] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:06.162492 6082 tablet_bootstrap.cc:654] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:06.163156 6083 log.cc:824] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:06.163156 6081 log.cc:824] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:06.163822 6082 log.cc:824] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:06.166270 6081 tablet_bootstrap.cc:492] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: No bootstrap required, opened a new log
I20260924 00:50:06.166532 6081 ts_tablet_manager.cc:1397] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 00:50:06.167419 6082 tablet_bootstrap.cc:492] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: No bootstrap required, opened a new log
I20260924 00:50:06.167650 6082 ts_tablet_manager.cc:1397] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 00:50:06.171536 6083 tablet_bootstrap.cc:492] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: No bootstrap required, opened a new log
I20260924 00:50:06.171875 6083 ts_tablet_manager.cc:1397] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260924 00:50:06.175351 6082 raft_consensus.cc:348] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.175449 6081 raft_consensus.cc:348] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.175765 6082 raft_consensus.cc:374] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:06.175915 6082 raft_consensus.cc:729] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f614b9268304dc19af77f4a397b9689, State: Initialized, Role: FOLLOWER
I20260924 00:50:06.175980 6081 raft_consensus.cc:374] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:06.176097 6081 raft_consensus.cc:729] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e890d986ad44e24a8167874b179a1d4, State: Initialized, Role: FOLLOWER
I20260924 00:50:06.176707 6082 consensus_queue.cc:260] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.176734 6081 consensus_queue.cc:260] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.177908 5932 heartbeater.cc:503] Master 127.0.222.62:42509 was elected leader, sending a full tablet report...
I20260924 00:50:06.179929 5800 heartbeater.cc:503] Master 127.0.222.62:42509 was elected leader, sending a full tablet report...
I20260924 00:50:06.180078 6082 ts_tablet_manager.cc:1428] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 00:50:06.181613 6081 ts_tablet_manager.cc:1428] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260924 00:50:06.182670 6083 raft_consensus.cc:348] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.183283 6083 raft_consensus.cc:374] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:06.183455 6083 raft_consensus.cc:729] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6237c77aac96458b8cb3bdbd4923d5e1, State: Initialized, Role: FOLLOWER
I20260924 00:50:06.184155 6083 consensus_queue.cc:260] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.185606 6063 heartbeater.cc:503] Master 127.0.222.62:42509 was elected leader, sending a full tablet report...
I20260924 00:50:06.186226 6083 ts_tablet_manager.cc:1428] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
W20260924 00:50:06.192853 5801 tablet.cc:2369] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:50:06.326745 6064 tablet.cc:2369] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:50:06.343483 6087 raft_consensus.cc:482] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:06.343870 6087 raft_consensus.cc:504] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.346035 6087 leader_election.cc:288] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6237c77aac96458b8cb3bdbd4923d5e1 (127.0.222.3:45985), 6e890d986ad44e24a8167874b179a1d4 (127.0.222.1:36599)
I20260924 00:50:06.353030 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.347514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:44855 (local address 127.0.222.1:36599)
0924 00:50:06.347683 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.347689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.348145 (+ 456us) server_negotiation.cc:407] Connection header received
0924 00:50:06.348307 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.348313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.348404 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.348500 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.352285 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.352291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.352296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.352319 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.352371 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.352386 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.352507 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.352660 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.352727 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:50:06.353027 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.346492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:44523 (local address 127.0.222.3:45985)
0924 00:50:06.346706 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.346714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.347255 (+ 541us) server_negotiation.cc:407] Connection header received
0924 00:50:06.347445 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.347451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.347524 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.347634 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.352285 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.352291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.352296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.352319 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.352364 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.352380 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.352514 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.352660 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.352727 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:50:06.353334 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.347301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.222.2:44855)
0924 00:50:06.348038 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.348058 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:06.348171 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:06.348592 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:06.348595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:06.348615 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:06.352011 (+ 3396us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:06.352015 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:06.352026 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:06.352087 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:06.352516 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:06.352522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:06.352529 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:50:06.352732 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 00:50:06.352800 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":18,"thread_start_us":403,"threads_started":1}
I20260924 00:50:06.353304 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.346403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:45985 (local address 127.0.222.2:44523)
0924 00:50:06.347123 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.347140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:06.347292 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:06.347798 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:06.347804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:06.347833 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:06.352001 (+ 4168us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:06.352005 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:06.352026 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:06.352087 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:06.352517 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:06.352522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:06.352529 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:50:06.352706 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 00:50:06.352783 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":447,"threads_started":1}
I20260924 00:50:06.354431 6017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594b5d456fd846bf9adb003b2a93aa24" candidate_uuid: "8f614b9268304dc19af77f4a397b9689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" is_pre_election: true
I20260924 00:50:06.354427 5755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594b5d456fd846bf9adb003b2a93aa24" candidate_uuid: "8f614b9268304dc19af77f4a397b9689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e890d986ad44e24a8167874b179a1d4" is_pre_election: true
I20260924 00:50:06.354899 6017 raft_consensus.cc:2454] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f614b9268304dc19af77f4a397b9689 in term 0.
I20260924 00:50:06.354897 5755 raft_consensus.cc:2454] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f614b9268304dc19af77f4a397b9689 in term 0.
I20260924 00:50:06.355545 5821 leader_election.cc:302] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6237c77aac96458b8cb3bdbd4923d5e1, 8f614b9268304dc19af77f4a397b9689; no voters:
I20260924 00:50:06.355985 6087 raft_consensus.cc:2781] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:06.356124 6087 raft_consensus.cc:482] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:06.356217 6087 raft_consensus.cc:3037] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:06.358482 6087 raft_consensus.cc:504] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.359086 6087 leader_election.cc:288] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [CANDIDATE]: Term 1 election: Requested vote from peers 6237c77aac96458b8cb3bdbd4923d5e1 (127.0.222.3:45985), 6e890d986ad44e24a8167874b179a1d4 (127.0.222.1:36599)
I20260924 00:50:06.359676 5755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594b5d456fd846bf9adb003b2a93aa24" candidate_uuid: "8f614b9268304dc19af77f4a397b9689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e890d986ad44e24a8167874b179a1d4"
I20260924 00:50:06.359639 6017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594b5d456fd846bf9adb003b2a93aa24" candidate_uuid: "8f614b9268304dc19af77f4a397b9689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6237c77aac96458b8cb3bdbd4923d5e1"
I20260924 00:50:06.359917 5755 raft_consensus.cc:3037] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:06.359901 6017 raft_consensus.cc:3037] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:06.363224 6017 raft_consensus.cc:2454] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f614b9268304dc19af77f4a397b9689 in term 1.
I20260924 00:50:06.363224 5755 raft_consensus.cc:2454] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f614b9268304dc19af77f4a397b9689 in term 1.
I20260924 00:50:06.363734 5821 leader_election.cc:302] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6237c77aac96458b8cb3bdbd4923d5e1, 8f614b9268304dc19af77f4a397b9689; no voters:
I20260924 00:50:06.364056 6087 raft_consensus.cc:2781] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:06.364532 6087 raft_consensus.cc:686] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 1 LEADER]: Becoming Leader. State: Replica: 8f614b9268304dc19af77f4a397b9689, State: Running, Role: LEADER
I20260924 00:50:06.365142 6087 consensus_queue.cc:237] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } }
I20260924 00:50:06.370556 5612 catalog_manager.cc:5495] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f614b9268304dc19af77f4a397b9689 (127.0.222.2). New cstate: current_term: 1 leader_uuid: "8f614b9268304dc19af77f4a397b9689" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f614b9268304dc19af77f4a397b9689" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 } health_report { overall_health: UNKNOWN } } }
W20260924 00:50:06.390702 5933 tablet.cc:2369] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:50:06.437690 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.435400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.0.222.2:33905)
0924 00:50:06.435573 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.435579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.435710 (+ 131us) server_negotiation.cc:407] Connection header received
0924 00:50:06.435970 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.435975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.436071 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.436160 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.437102 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.437107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.437110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.437125 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.437157 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.437168 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.437318 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.437384 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.437445 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:50:06.446091 6017 raft_consensus.cc:1261] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 1 FOLLOWER]: Refusing update from remote peer 8f614b9268304dc19af77f4a397b9689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:06.446092 5755 raft_consensus.cc:1261] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 1 FOLLOWER]: Refusing update from remote peer 8f614b9268304dc19af77f4a397b9689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:06.447198 6087 consensus_queue.cc:1035] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6237c77aac96458b8cb3bdbd4923d5e1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:06.448047 6092 consensus_queue.cc:1035] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e890d986ad44e24a8167874b179a1d4" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:06.466291 6095 mvcc.cc:205] Tried to move back new op lower bound from 7332704282391842816 to 7332704282076053504. Current Snapshot: MvccSnapshot[applied={T|T < 7332704282391842816}]
I20260924 00:50:06.491539 5612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:06.491873 5612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:06.496788 5612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee8f5a0f1668412ba5cec6d468855f1a: Sending DeleteTablet for 3 replicas of tablet 594b5d456fd846bf9adb003b2a93aa24
I20260924 00:50:06.498617 5866 tablet_service.cc:1514] Processing DeleteTablet for tablet 594b5d456fd846bf9adb003b2a93aa24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:06 UTC) from {username='slave'} at 127.0.0.1:43988
I20260924 00:50:06.499060 5997 tablet_service.cc:1514] Processing DeleteTablet for tablet 594b5d456fd846bf9adb003b2a93aa24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:06 UTC) from {username='slave'} at 127.0.0.1:45866
I20260924 00:50:06.500576 5735 tablet_service.cc:1514] Processing DeleteTablet for tablet 594b5d456fd846bf9adb003b2a93aa24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:06 UTC) from {username='slave'} at 127.0.0.1:41584
I20260924 00:50:06.505407 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 5672
I20260924 00:50:06.507366 6106 tablet_replica.cc:330] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: stopping tablet replica
I20260924 00:50:06.507855 6106 raft_consensus.cc:2229] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:06.508401 6106 raft_consensus.cc:2258] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:06.509230 6107 tablet_replica.cc:330] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: stopping tablet replica
I20260924 00:50:06.509675 6107 raft_consensus.cc:2229] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:06.510113 6107 raft_consensus.cc:2258] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:06.514844 6106 ts_tablet_manager.cc:1905] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:06.521847 6107 ts_tablet_manager.cc:1905] T 594b5d456fd846bf9adb003b2a93aa24 P 6e890d986ad44e24a8167874b179a1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:06.523051 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.506847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.0.222.1:36599)
0924 00:50:06.507043 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.507051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.520178 (+ 13127us) server_negotiation.cc:407] Connection header received
0924 00:50:06.520700 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.520706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.520812 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.520899 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.522233 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.522239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.522244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.522277 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.522315 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.522332 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.522437 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.522659 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.522726 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:50:06.525825 5795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:06.525849 6105 tablet_replica.cc:330] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: stopping tablet replica
I20260924 00:50:06.526463 6105 raft_consensus.cc:2229] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:50:06.527184 6105 raft_consensus.cc:2258] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:06.528618 6106 ts_tablet_manager.cc:1918] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:06.529008 6106 log.cc:1196] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/594b5d456fd846bf9adb003b2a93aa24
I20260924 00:50:06.529745 6106 ts_tablet_manager.cc:1939] T 594b5d456fd846bf9adb003b2a93aa24 P 6237c77aac96458b8cb3bdbd4923d5e1: Deleting consensus metadata
I20260924 00:50:06.530194 6105 ts_tablet_manager.cc:1905] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:06.531711 5598 catalog_manager.cc:4841] TS 6237c77aac96458b8cb3bdbd4923d5e1 (127.0.222.3:45985): tablet 594b5d456fd846bf9adb003b2a93aa24 (table test-table [id=dcffaf72c28840be86c84d70db2b6748]) successfully deleted
I20260924 00:50:06.538223 6105 ts_tablet_manager.cc:1918] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:06.538441 6105 log.cc:1196] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/594b5d456fd846bf9adb003b2a93aa24
I20260924 00:50:06.539004 6105 ts_tablet_manager.cc:1939] T 594b5d456fd846bf9adb003b2a93aa24 P 8f614b9268304dc19af77f4a397b9689: Deleting consensus metadata
I20260924 00:50:06.541090 5598 catalog_manager.cc:4841] TS 8f614b9268304dc19af77f4a397b9689 (127.0.222.2:33905): tablet 594b5d456fd846bf9adb003b2a93aa24 (table test-table [id=dcffaf72c28840be86c84d70db2b6748]) successfully deleted
I20260924 00:50:06.621889 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 5672
W20260924 00:50:06.638130 5598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:36599 (error 108)
I20260924 00:50:06.644832 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 5804
I20260924 00:50:06.648887 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.646009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.222.2:33905)
0924 00:50:06.646211 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.646217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.646240 (+ 23us) server_negotiation.cc:407] Connection header received
0924 00:50:06.646336 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.646345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.646426 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.646555 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.647807 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.647813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.647818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.647845 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.647886 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.647903 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.648036 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.648495 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.648565 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 00:50:06.649950 5927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:06.657843 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.654932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.0.1:38316)
0924 00:50:06.657464 (+ 2532us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.657640 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2386,"thread_start_us":2188,"threads_started":1}
W20260924 00:50:06.658181 5598 catalog_manager.cc:4584] TS 6e890d986ad44e24a8167874b179a1d4 (127.0.222.1:36599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 594b5d456fd846bf9adb003b2a93aa24: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36599: connect: Connection refused (error 111)
I20260924 00:50:06.712234 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.711687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.0.1:38318)
0924 00:50:06.711949 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.712088 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 00:50:06.713515 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.713044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.0.1:38324)
0924 00:50:06.713253 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.713347 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 00:50:06.765599 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 5804
I20260924 00:50:06.773947 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.773481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.0.1:38332)
0924 00:50:06.773696 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.773790 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 00:50:06.775122 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.774673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.0.1:38344)
0924 00:50:06.774860 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.774977 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 00:50:06.780998 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 5935
I20260924 00:50:06.783916 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.781810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.0.222.3:45985)
0924 00:50:06.781967 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:06.781971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:06.782112 (+ 141us) server_negotiation.cc:407] Connection header received
0924 00:50:06.782186 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:06.782190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:06.782293 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:06.782386 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:06.783316 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:06.783320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:06.783323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:06.783339 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:06.783375 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:06.783387 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:06.783503 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:06.783575 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 00:50:06.783623 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:50:06.784549 6058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:06.856276 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.855728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.0.1:38358)
0924 00:50:06.855977 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.856089 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 00:50:06.857570 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:06.857032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36599 (local address 127.0.0.1:38360)
0924 00:50:06.857305 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:06.857411 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 00:50:06.876816 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 5935
I20260924 00:50:06.890306 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 5581
I20260924 00:50:06.891588 5642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:06.998219 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 5581
2026-09-24T00:50:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2853 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 00:50:07.036592 888 mini_oidc.cc:152] Starting JWKS server
I20260924 00:50:07.039327 888 webserver.cc:460] Webserver started at https://127.0.0.1:34575/ using document root <none> and password file <none>
I20260924 00:50:07.039626 888 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:50:07.040328 888 webserver.cc:460] Webserver started at http://127.0.0.1:44585/ using document root <none> and password file <none>
2026-09-24T00:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:07Z Disabled control of system clock
I20260924 00:50:07.066565 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41879
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41879
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34575/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/testchainca.cert with env {}
W20260924 00:50:07.400031 6120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:07.400318 6120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:07.400413 6120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:07.407749 6120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:07.407878 6120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:07.407915 6120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:07.407972 6120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:07.408015 6120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:50:07.408074 6120 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:50:07.408116 6120 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34575/jwks.json
I20260924 00:50:07.418603 6120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34575/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:07.419909 6120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:07.421166 6120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:07.432812 6125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:07.432780 6126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:07.432850 6128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:07.433640 6120 server_base.cc:1034] running on GCE node
I20260924 00:50:07.434514 6120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:07.435981 6120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:07.437209 6120 hybrid_clock.cc:648] HybridClock initialized: now 1790211007437127 us; error 59 us; skew 500 ppm
I20260924 00:50:07.437598 6120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:07.440150 6120 webserver.cc:460] Webserver started at http://127.0.222.62:34303/ using document root <none> and password file <none>
I20260924 00:50:07.440902 6120 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:07.441008 6120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:07.441351 6120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:07.444312 6120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "ca4195f4602d4aa888acaf2b1533dd5c"
format_stamp: "Formatted at 2026-09-24 00:50:07 on dist-test-slave-rtvz"
server_key: "769558cdf33ec57cc1035283159028e9"
server_key_iv: "f7b05f4da77486d9b832f9fc3f7dbd63"
server_key_version: "encryptionkey@0"
I20260924 00:50:07.445139 6120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "ca4195f4602d4aa888acaf2b1533dd5c"
format_stamp: "Formatted at 2026-09-24 00:50:07 on dist-test-slave-rtvz"
server_key: "769558cdf33ec57cc1035283159028e9"
server_key_iv: "f7b05f4da77486d9b832f9fc3f7dbd63"
server_key_version: "encryptionkey@0"
I20260924 00:50:07.450590 6120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:50:07.454324 6134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:07.455879 6120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:50:07.456070 6120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "ca4195f4602d4aa888acaf2b1533dd5c"
format_stamp: "Formatted at 2026-09-24 00:50:07 on dist-test-slave-rtvz"
server_key: "769558cdf33ec57cc1035283159028e9"
server_key_iv: "f7b05f4da77486d9b832f9fc3f7dbd63"
server_key_version: "encryptionkey@0"
I20260924 00:50:07.456288 6120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:07.470470 6120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:07.471379 6120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:07.471681 6120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:07.489279 6120 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41879
I20260924 00:50:07.489310 6186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41879 every 8 connection(s)
I20260924 00:50:07.490904 6120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:07.494884 6187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:07.496253 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 6120
I20260924 00:50:07.496425 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:07.497047 888 external_mini_cluster.cc:1428] Setting key 5cbf72e7d914ef56eb2978a93fba02c3
I20260924 00:50:07.507485 6187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c: Bootstrap starting.
I20260924 00:50:07.509869 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:07.498840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.0.222.62:41879)
0924 00:50:07.499788 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:07.499799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:07.499839 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:50:07.501429 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:07.501453 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:07.501922 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:07.502273 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:07.502882 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:07.504155 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:07.505538 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:07.506201 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:07.506376 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:07.507567 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:07.507577 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:07.507596 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:07.508160 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:07.508242 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:07.508264 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:07.508368 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:07.508577 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 00:50:07.508892 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":316,"threads_started":1}
I20260924 00:50:07.510811 6187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:07.512178 6187 log.cc:824] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:07.514048 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:50:07.515426 6187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c: No bootstrap required, opened a new log
I20260924 00:50:07.523142 6187 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4195f4602d4aa888acaf2b1533dd5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41879 } }
I20260924 00:50:07.523556 6187 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:07.523633 6187 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4195f4602d4aa888acaf2b1533dd5c, State: Initialized, Role: FOLLOWER
I20260924 00:50:07.524161 6187 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4195f4602d4aa888acaf2b1533dd5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41879 } }
I20260924 00:50:07.524365 6187 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:07.524475 6187 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:07.524601 6187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:07.527273 6187 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4195f4602d4aa888acaf2b1533dd5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41879 } }
I20260924 00:50:07.527868 6187 leader_election.cc:302] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca4195f4602d4aa888acaf2b1533dd5c; no voters:
I20260924 00:50:07.528453 6187 leader_election.cc:288] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:07.528587 6192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:07.529613 6192 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [term 1 LEADER]: Becoming Leader. State: Replica: ca4195f4602d4aa888acaf2b1533dd5c, State: Running, Role: LEADER
I20260924 00:50:07.530120 6192 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4195f4602d4aa888acaf2b1533dd5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41879 } }
I20260924 00:50:07.530656 6187 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:07.533010 6193 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca4195f4602d4aa888acaf2b1533dd5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4195f4602d4aa888acaf2b1533dd5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41879 } } }
I20260924 00:50:07.533418 6193 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:07.533111 6194 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4195f4602d4aa888acaf2b1533dd5c. Latest consensus state: current_term: 1 leader_uuid: "ca4195f4602d4aa888acaf2b1533dd5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4195f4602d4aa888acaf2b1533dd5c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41879 } } }
I20260924 00:50:07.533665 6194 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:07.538470 6197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:07.544253 6197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:07.554198 6197 catalog_manager.cc:1347] Generated new cluster ID: 0b80020452d444d09ca46081b370ccb1
I20260924 00:50:07.554330 6197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:07.584064 6197 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:07.585596 6197 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:07.594630 6197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca4195f4602d4aa888acaf2b1533dd5c: Generated new TSK 0
I20260924 00:50:07.595534 6197 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 00:50:07.640774 6217 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 00:50:07.641012 6217 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 00:50:07.641937 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:07.611915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.0.222.62:41879)
0924 00:50:07.612135 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:07.612142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:07.612728 (+ 586us) server_negotiation.cc:407] Connection header received
0924 00:50:07.613099 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:07.613103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:07.613175 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:07.613331 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:50:07.614082 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:07.614890 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:07.617024 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:07.617730 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:07.617828 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:07.618559 (+ 731us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:50:07.641344 (+ 22785us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 00:50:07.641681 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 00:50:07.642060 6190 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 00:50:07.643970 6216 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:07.611880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41879 (local address 127.0.0.1:40922)
0924 00:50:07.612586 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:07.612611 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:07.612839 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:07.613379 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:07.613384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:07.613398 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:50:07.613909 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:07.613925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:07.615056 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:07.615063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:07.616875 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:07.616892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:07.617942 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:07.617950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:07.618144 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:07.618237 (+ 93us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:50:07.641652 (+ 23415us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 00:50:07.642043 (+ 391us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.222.62:41879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":328,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.222.62:41879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 00:50:07.645804 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 6120
I20260924 00:50:07.647006 6182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:07.744642 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 6120
2026-09-24T00:50:07Z chronyd exiting
I20260924 00:50:07.791666 888 test_util.cc:175] -----------------------------------------------
I20260924 00:50:07.791791 888 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790210962623352-888-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (759 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 00:50:07.795601 888 mini_oidc.cc:152] Starting JWKS server
I20260924 00:50:07.797417 888 webserver.cc:460] Webserver started at https://127.0.0.1:45595/ using document root <none> and password file <none>
I20260924 00:50:07.797652 888 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:50:07.798256 888 webserver.cc:460] Webserver started at http://127.0.0.1:42375/ using document root <none> and password file <none>
2026-09-24T00:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:07Z Disabled control of system clock
I20260924 00:50:07.823734 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46597
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:45263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:46597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45595/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/testchainca.cert with env {}
W20260924 00:50:08.154068 6224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:08.154392 6224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:08.154505 6224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:08.162087 6224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:08.162213 6224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:08.162282 6224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:08.162358 6224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:08.162405 6224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:50:08.162487 6224 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:50:08.162544 6224 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45595/jwks.json
I20260924 00:50:08.172915 6224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:46597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45595/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:08.174333 6224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:08.175683 6224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:08.188309 6232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:08.188356 6230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:08.188378 6229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:08.189272 6224 server_base.cc:1034] running on GCE node
I20260924 00:50:08.190192 6224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:08.191888 6224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:08.193101 6224 hybrid_clock.cc:648] HybridClock initialized: now 1790211008193045 us; error 48 us; skew 500 ppm
I20260924 00:50:08.193554 6224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:08.195923 6224 webserver.cc:460] Webserver started at http://127.0.222.62:34949/ using document root <none> and password file <none>
I20260924 00:50:08.196655 6224 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:08.196759 6224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:08.197078 6224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:08.200215 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "ed532ad63a22465bbca71222f229cef1"
format_stamp: "Formatted at 2026-09-24 00:50:08 on dist-test-slave-rtvz"
server_key: "248d6855100176292a49fca29da2b555"
server_key_iv: "36cf645dbb0149c066dabf9fb76ad524"
server_key_version: "encryptionkey@0"
I20260924 00:50:08.201046 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "ed532ad63a22465bbca71222f229cef1"
format_stamp: "Formatted at 2026-09-24 00:50:08 on dist-test-slave-rtvz"
server_key: "248d6855100176292a49fca29da2b555"
server_key_iv: "36cf645dbb0149c066dabf9fb76ad524"
server_key_version: "encryptionkey@0"
I20260924 00:50:08.206473 6224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:50:08.210253 6238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:08.211755 6224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:50:08.211937 6224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "ed532ad63a22465bbca71222f229cef1"
format_stamp: "Formatted at 2026-09-24 00:50:08 on dist-test-slave-rtvz"
server_key: "248d6855100176292a49fca29da2b555"
server_key_iv: "36cf645dbb0149c066dabf9fb76ad524"
server_key_version: "encryptionkey@0"
I20260924 00:50:08.212149 6224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:08.231419 6224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:08.232339 6224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:08.232663 6224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:08.250717 6224 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:46597
I20260924 00:50:08.250735 6289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:46597 every 8 connection(s)
I20260924 00:50:08.252104 6224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:08.252493 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 6224
I20260924 00:50:08.252686 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:08.253499 888 external_mini_cluster.cc:1428] Setting key 0ea7427f3a2b5c030063d688b7889f7f
I20260924 00:50:08.256722 6290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:08.267202 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:08.255549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.0.222.62:46597)
0924 00:50:08.256591 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:08.256602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:08.256645 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:50:08.258313 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:08.258336 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:08.258805 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:08.259193 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:08.259778 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:08.261113 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:08.262422 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:08.263105 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:08.263275 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:08.264610 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:08.264618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:08.264637 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:08.265295 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:08.265428 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:08.265451 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:08.265566 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:08.265807 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 00:50:08.266170 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":358,"threads_started":1}
I20260924 00:50:08.267817 6290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1: Bootstrap starting.
I20260924 00:50:08.271698 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:50:08.271773 6290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:08.272876 6290 log.cc:824] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:08.276397 6290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1: No bootstrap required, opened a new log
I20260924 00:50:08.284565 6290 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed532ad63a22465bbca71222f229cef1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46597 } }
I20260924 00:50:08.284977 6290 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:08.285058 6290 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed532ad63a22465bbca71222f229cef1, State: Initialized, Role: FOLLOWER
I20260924 00:50:08.285643 6290 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed532ad63a22465bbca71222f229cef1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46597 } }
I20260924 00:50:08.285874 6290 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:08.285987 6290 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:08.286130 6290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:08.288362 6290 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed532ad63a22465bbca71222f229cef1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46597 } }
I20260924 00:50:08.288877 6290 leader_election.cc:302] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed532ad63a22465bbca71222f229cef1; no voters:
I20260924 00:50:08.289561 6290 leader_election.cc:288] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:08.289721 6295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:08.290591 6295 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [term 1 LEADER]: Becoming Leader. State: Replica: ed532ad63a22465bbca71222f229cef1, State: Running, Role: LEADER
I20260924 00:50:08.291124 6295 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed532ad63a22465bbca71222f229cef1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46597 } }
I20260924 00:50:08.291872 6290 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:08.293892 6297 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed532ad63a22465bbca71222f229cef1. Latest consensus state: current_term: 1 leader_uuid: "ed532ad63a22465bbca71222f229cef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed532ad63a22465bbca71222f229cef1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46597 } } }
I20260924 00:50:08.294461 6297 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:08.299479 6296 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed532ad63a22465bbca71222f229cef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed532ad63a22465bbca71222f229cef1" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46597 } } }
I20260924 00:50:08.299875 6296 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:08.301837 6304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:08.307965 6304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:08.318229 6304 catalog_manager.cc:1347] Generated new cluster ID: 8f5c75ffadcb4e6cafb41babd86dd15a
I20260924 00:50:08.318369 6304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:08.332935 6304 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:08.334972 6304 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:08.343660 6304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed532ad63a22465bbca71222f229cef1: Generated new TSK 0
I20260924 00:50:08.344887 6304 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 00:50:08.387012 6320 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 00:50:08.387128 6320 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 00:50:08.388509 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:08.366758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.0.222.62:46597)
0924 00:50:08.366985 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:08.366993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:08.367433 (+ 440us) server_negotiation.cc:407] Connection header received
0924 00:50:08.367668 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:08.367674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:08.367776 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:08.367975 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:50:08.368527 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:08.369718 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:08.371361 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:08.372113 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:08.372256 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:08.372852 (+ 596us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:50:08.387716 (+ 14864us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 00:50:08.388171 (+ 455us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 00:50:08.388628 6319 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:08.366548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46597 (local address 127.0.0.1:60226)
0924 00:50:08.367312 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:08.367331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:08.367475 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:08.368018 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:08.368022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:08.368033 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:50:08.368337 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:08.368348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:08.369918 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:08.369926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:08.371153 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:08.371170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:08.372346 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:08.372352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:08.372512 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:08.372553 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:50:08.388030 (+ 15477us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 00:50:08.388312 (+ 282us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.222.62:46597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":446,"threads_started":1}
W20260924 00:50:08.388650 6293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45595/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 00:50:08.390595 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 6224
I20260924 00:50:08.391851 6285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:08.486719 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 6224
2026-09-24T00:50:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (743 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 00:50:08.539445 888 mini_oidc.cc:152] Starting JWKS server
I20260924 00:50:08.541433 888 webserver.cc:460] Webserver started at https://127.0.0.1:39479/ using document root <none> and password file <none>
I20260924 00:50:08.541666 888 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:50:08.542362 888 webserver.cc:460] Webserver started at http://127.0.0.1:38063/ using document root <none> and password file <none>
2026-09-24T00:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:08Z Disabled control of system clock
I20260924 00:50:08.567787 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34191
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:45743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:34191
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39479/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 00:50:08.897573 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:08.897912 6327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:08.898051 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:08.905452 6327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:08.905580 6327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:08.905642 6327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:08.905728 6327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:08.905788 6327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:50:08.905874 6327 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:50:08.905925 6327 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39479/jwks.json
I20260924 00:50:08.915324 6327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:34191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39479/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:08.916745 6327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:08.918653 6327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:08.931160 6332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:08.931242 6335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:08.931176 6333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:08.933271 6327 server_base.cc:1034] running on GCE node
I20260924 00:50:08.934186 6327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:08.935740 6327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:08.936964 6327 hybrid_clock.cc:648] HybridClock initialized: now 1790211008936909 us; error 42 us; skew 500 ppm
I20260924 00:50:08.937399 6327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:08.939944 6327 webserver.cc:460] Webserver started at http://127.0.222.62:34627/ using document root <none> and password file <none>
I20260924 00:50:08.940651 6327 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:08.940761 6327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:08.941087 6327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:08.944250 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "f941bed05b154a7ab3404d414120855f"
format_stamp: "Formatted at 2026-09-24 00:50:08 on dist-test-slave-rtvz"
server_key: "703b1fc9c3d3e6fa196e3ddfb5ed9003"
server_key_iv: "f3033d78e68599eee7bddee1018a6488"
server_key_version: "encryptionkey@0"
I20260924 00:50:08.945111 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "f941bed05b154a7ab3404d414120855f"
format_stamp: "Formatted at 2026-09-24 00:50:08 on dist-test-slave-rtvz"
server_key: "703b1fc9c3d3e6fa196e3ddfb5ed9003"
server_key_iv: "f3033d78e68599eee7bddee1018a6488"
server_key_version: "encryptionkey@0"
I20260924 00:50:08.950803 6327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 00:50:08.954866 6341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:08.956416 6327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:50:08.956609 6327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "f941bed05b154a7ab3404d414120855f"
format_stamp: "Formatted at 2026-09-24 00:50:08 on dist-test-slave-rtvz"
server_key: "703b1fc9c3d3e6fa196e3ddfb5ed9003"
server_key_iv: "f3033d78e68599eee7bddee1018a6488"
server_key_version: "encryptionkey@0"
I20260924 00:50:08.956858 6327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:08.981549 6327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:08.982497 6327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:08.982817 6327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:09.001350 6327 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:34191
I20260924 00:50:09.001394 6395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:34191 every 8 connection(s)
I20260924 00:50:09.002744 6327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:09.006584 6396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:09.007316 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 6327
I20260924 00:50:09.007513 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:09.008365 888 external_mini_cluster.cc:1428] Setting key 5a1135e3e9f9ccd0334417f59fc7ba29
I20260924 00:50:09.020869 6396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f: Bootstrap starting.
I20260924 00:50:09.022667 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:09.010527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.0.222.62:34191)
0924 00:50:09.011572 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:09.011584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:09.011629 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:50:09.013292 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:09.013316 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:09.013881 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:09.014242 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:09.014917 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.016233 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.018077 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.018827 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.018973 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:09.020304 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:09.020312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:09.020327 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:09.020890 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:09.020981 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:09.021009 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:09.021131 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:09.021389 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 00:50:09.021706 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":370,"threads_started":1}
I20260924 00:50:09.024120 6396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:09.025179 6396 log.cc:824] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:09.026345 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:50:09.027891 6396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f: No bootstrap required, opened a new log
I20260924 00:50:09.035048 6396 raft_consensus.cc:348] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f941bed05b154a7ab3404d414120855f" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34191 } }
I20260924 00:50:09.035441 6396 raft_consensus.cc:374] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:09.035543 6396 raft_consensus.cc:729] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f941bed05b154a7ab3404d414120855f, State: Initialized, Role: FOLLOWER
I20260924 00:50:09.036139 6396 consensus_queue.cc:260] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f941bed05b154a7ab3404d414120855f" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34191 } }
I20260924 00:50:09.036362 6396 raft_consensus.cc:388] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:09.036504 6396 raft_consensus.cc:482] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:09.036649 6396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:09.039165 6396 raft_consensus.cc:504] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f941bed05b154a7ab3404d414120855f" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34191 } }
I20260924 00:50:09.039707 6396 leader_election.cc:302] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f941bed05b154a7ab3404d414120855f; no voters:
I20260924 00:50:09.040268 6396 leader_election.cc:288] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:09.040446 6401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:09.041370 6401 raft_consensus.cc:686] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [term 1 LEADER]: Becoming Leader. State: Replica: f941bed05b154a7ab3404d414120855f, State: Running, Role: LEADER
I20260924 00:50:09.041996 6401 consensus_queue.cc:237] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f941bed05b154a7ab3404d414120855f" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34191 } }
I20260924 00:50:09.042505 6396 sys_catalog.cc:564] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:09.044721 6403 sys_catalog.cc:455] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f941bed05b154a7ab3404d414120855f. Latest consensus state: current_term: 1 leader_uuid: "f941bed05b154a7ab3404d414120855f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f941bed05b154a7ab3404d414120855f" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34191 } } }
I20260924 00:50:09.045135 6403 sys_catalog.cc:458] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:09.045801 6402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f941bed05b154a7ab3404d414120855f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f941bed05b154a7ab3404d414120855f" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34191 } } }
I20260924 00:50:09.046240 6402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:09.046882 6406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:09.052364 6406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:09.062330 6406 catalog_manager.cc:1347] Generated new cluster ID: f261baaa6b5a454caf16bb7451e87cff
I20260924 00:50:09.062462 6406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:09.096922 6406 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:09.098315 6406 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:09.110342 6406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f941bed05b154a7ab3404d414120855f: Generated new TSK 0
I20260924 00:50:09.111251 6406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:50:09.135353 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:09.126282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.222.62:34191)
0924 00:50:09.126475 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:09.126481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:09.126803 (+ 322us) server_negotiation.cc:407] Connection header received
0924 00:50:09.127034 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:09.127040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:09.127141 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:09.127320 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:50:09.127892 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.129030 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.130651 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.131370 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.131528 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:09.132137 (+ 609us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:50:09.134672 (+ 2535us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39479/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 00:50:09.135030 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40888: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39479/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 00:50:09.135490 6399 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40888: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39479/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 00:50:09.137275 6425 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:09.126114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34191 (local address 127.0.0.1:40888)
0924 00:50:09.126699 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:09.126715 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:09.126845 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:09.127367 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:09.127371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:09.127382 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:50:09.127671 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:09.127680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.129271 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.129276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:09.130481 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:09.130497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.131604 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.131613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:09.131774 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:09.131815 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:50:09.134926 (+ 3111us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39479/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 00:50:09.136803 (+ 1877us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.222.62:34191: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39479/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":1509,"thread_start_us":303,"threads_started":1}
I20260924 00:50:09.138491 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 6327
I20260924 00:50:09.139858 6390 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 00:50:09.146988 6426 webserver.cc:533] Webserver: sslize failed:
E20260924 00:50:09.147089 6426 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 00:50:09.234309 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 6327
2026-09-24T00:50:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T00:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:09Z Disabled control of system clock
I20260924 00:50:09.312196 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42721
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42721
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/testchainca.cert with env {}
W20260924 00:50:09.642642 6431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:09.642971 6431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:09.643059 6431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:09.650528 6431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:09.650652 6431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:09.650689 6431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:09.650738 6431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:09.650796 6431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:50:09.650882 6431 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:50:09.650938 6431 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 00:50:09.661590 6431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:09.662911 6431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:09.664381 6431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:09.676339 6437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:09.676393 6436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:09.676425 6439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:09.677014 6431 server_base.cc:1034] running on GCE node
I20260924 00:50:09.677855 6431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:09.679352 6431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:09.680573 6431 hybrid_clock.cc:648] HybridClock initialized: now 1790211009680518 us; error 51 us; skew 500 ppm
I20260924 00:50:09.680936 6431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:09.683286 6431 webserver.cc:460] Webserver started at http://127.0.222.62:34811/ using document root <none> and password file <none>
I20260924 00:50:09.684046 6431 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:09.684157 6431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:09.684468 6431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:09.687575 6431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "92c4a94738be4deeae56f0d6a99cf28a"
format_stamp: "Formatted at 2026-09-24 00:50:09 on dist-test-slave-rtvz"
server_key: "c1075f01417ef5460170f055fd617b23"
server_key_iv: "bf5ea74c6c298ed9ca97f230077341eb"
server_key_version: "encryptionkey@0"
I20260924 00:50:09.688468 6431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "92c4a94738be4deeae56f0d6a99cf28a"
format_stamp: "Formatted at 2026-09-24 00:50:09 on dist-test-slave-rtvz"
server_key: "c1075f01417ef5460170f055fd617b23"
server_key_iv: "bf5ea74c6c298ed9ca97f230077341eb"
server_key_version: "encryptionkey@0"
I20260924 00:50:09.694200 6431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 00:50:09.698333 6445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:09.700080 6431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260924 00:50:09.700286 6431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "92c4a94738be4deeae56f0d6a99cf28a"
format_stamp: "Formatted at 2026-09-24 00:50:09 on dist-test-slave-rtvz"
server_key: "c1075f01417ef5460170f055fd617b23"
server_key_iv: "bf5ea74c6c298ed9ca97f230077341eb"
server_key_version: "encryptionkey@0"
I20260924 00:50:09.700631 6431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:09.738408 6431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:09.739307 6431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:09.739616 6431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:09.757426 6431 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42721
I20260924 00:50:09.757449 6497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42721 every 8 connection(s)
I20260924 00:50:09.758867 6431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:09.762856 6498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:09.763236 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 6431
I20260924 00:50:09.763453 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:09.764259 888 external_mini_cluster.cc:1428] Setting key eb2d752b6b54df6c2b5ada7fd74b5109
I20260924 00:50:09.777052 6498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a: Bootstrap starting.
I20260924 00:50:09.778601 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:09.766257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.0.222.62:42721)
0924 00:50:09.767288 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:09.767299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:09.767347 (+ 48us) server_negotiation.cc:407] Connection header received
0924 00:50:09.769164 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:09.769192 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:09.769786 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:09.770145 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:09.770857 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.772139 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.773971 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.774726 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.774872 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:09.776094 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:09.776102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:09.776118 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:09.776705 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:09.776806 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:09.776829 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:09.776918 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:09.777158 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 00:50:09.777569 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":369,"threads_started":1}
I20260924 00:50:09.780304 6498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:09.781466 6498 log.cc:824] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:09.782423 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:50:09.784768 6498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a: No bootstrap required, opened a new log
I20260924 00:50:09.791949 6498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c4a94738be4deeae56f0d6a99cf28a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42721 } }
I20260924 00:50:09.792348 6498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:09.792424 6498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c4a94738be4deeae56f0d6a99cf28a, State: Initialized, Role: FOLLOWER
I20260924 00:50:09.792953 6498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c4a94738be4deeae56f0d6a99cf28a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42721 } }
I20260924 00:50:09.793174 6498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:09.793372 6498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:09.793565 6498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:09.795714 6498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c4a94738be4deeae56f0d6a99cf28a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42721 } }
I20260924 00:50:09.796213 6498 leader_election.cc:302] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92c4a94738be4deeae56f0d6a99cf28a; no voters:
I20260924 00:50:09.796847 6498 leader_election.cc:288] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:09.796988 6503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:09.798070 6503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [term 1 LEADER]: Becoming Leader. State: Replica: 92c4a94738be4deeae56f0d6a99cf28a, State: Running, Role: LEADER
I20260924 00:50:09.798604 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c4a94738be4deeae56f0d6a99cf28a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42721 } }
I20260924 00:50:09.799034 6498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:09.801604 6504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92c4a94738be4deeae56f0d6a99cf28a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c4a94738be4deeae56f0d6a99cf28a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42721 } } }
I20260924 00:50:09.801976 6504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:09.802292 6505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c4a94738be4deeae56f0d6a99cf28a. Latest consensus state: current_term: 1 leader_uuid: "92c4a94738be4deeae56f0d6a99cf28a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92c4a94738be4deeae56f0d6a99cf28a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42721 } } }
I20260924 00:50:09.802615 6505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:09.806161 6509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:09.811563 6509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:09.821949 6509 catalog_manager.cc:1347] Generated new cluster ID: a84e0cc39c99464bbdff52568544efe1
I20260924 00:50:09.822079 6509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:09.844409 6509 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:09.845752 6509 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:09.858179 6509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92c4a94738be4deeae56f0d6a99cf28a: Generated new TSK 0
I20260924 00:50:09.859081 6509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:50:09.977350 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:09.880099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.0.222.62:42721)
0924 00:50:09.880301 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:09.880307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:09.880726 (+ 419us) server_negotiation.cc:407] Connection header received
0924 00:50:09.881000 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:09.881007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:09.881121 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:09.881307 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:50:09.881780 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.883026 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.884827 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.885533 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.885668 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:09.886338 (+ 670us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:50:09.976657 (+ 90319us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 00:50:09.977061 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42202: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 00:50:09.977510 6501 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42202: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 00:50:09.977727 6527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:09.879984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42721 (local address 127.0.0.1:42202)
0924 00:50:09.880619 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:09.880638 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:09.880797 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:09.881339 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:09.881344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:09.881357 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:50:09.881623 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:09.881633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.883166 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.883172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:09.884673 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:09.884689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:09.885727 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:09.885734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:09.885983 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:09.886034 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:50:09.976956 (+ 90922us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 00:50:09.977316 (+ 360us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.222.62:42721: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":306,"threads_started":1}
I20260924 00:50:09.979406 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 6431
I20260924 00:50:09.980855 6493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:10.084923 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 6431
2026-09-24T00:50:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (830 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (416 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (418 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:50:11 dist-test-slave-rtvz krb5kdc[6543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:50:11 dist-test-slave-rtvz krb5kdc[6543](info): set up 2 sockets
Sep 24 00:50:11 dist-test-slave-rtvz krb5kdc[6543](info): commencing operation
krb5kdc: starting...
W20260924 00:50:11.932837 888 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:50:12 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:50:12.138224 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44207
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:12.460784 6556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:12.461150 6556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:12.461311 6556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:12.468607 6556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:12.468746 6556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:12.468832 6556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:12.468909 6556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:12.468971 6556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:12.479041 6556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:12.480403 6556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:12.481765 6556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:12.493644 6562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:12.493677 6561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:12.493700 6564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:12.494517 6556 server_base.cc:1034] running on GCE node
I20260924 00:50:12.495358 6556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:12.496906 6556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:12.498148 6556 hybrid_clock.cc:648] HybridClock initialized: now 1790211012498061 us; error 78 us; skew 500 ppm
I20260924 00:50:12.498534 6556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:12 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:12.507503 6556 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:50:12.509143 6556 webserver.cc:460] Webserver started at http://127.0.222.62:45373/ using document root <none> and password file <none>
I20260924 00:50:12.509888 6556 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:12.510012 6556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:12.510319 6556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:12.513459 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "9ad73d3d4c964d38a3100dfa2a89c0af"
format_stamp: "Formatted at 2026-09-24 00:50:12 on dist-test-slave-rtvz"
server_key: "b4025af48338bdae0e0bcce27d4210bd"
server_key_iv: "a74942b7ef80a50192eba34d6422cc1a"
server_key_version: "encryptionkey@0"
I20260924 00:50:12.514242 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "9ad73d3d4c964d38a3100dfa2a89c0af"
format_stamp: "Formatted at 2026-09-24 00:50:12 on dist-test-slave-rtvz"
server_key: "b4025af48338bdae0e0bcce27d4210bd"
server_key_iv: "a74942b7ef80a50192eba34d6422cc1a"
server_key_version: "encryptionkey@0"
I20260924 00:50:12.519564 6556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 00:50:12.523286 6571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:12.524734 6556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:12.524945 6556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "9ad73d3d4c964d38a3100dfa2a89c0af"
format_stamp: "Formatted at 2026-09-24 00:50:12 on dist-test-slave-rtvz"
server_key: "b4025af48338bdae0e0bcce27d4210bd"
server_key_iv: "a74942b7ef80a50192eba34d6422cc1a"
server_key_version: "encryptionkey@0"
I20260924 00:50:12.525148 6556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:12.536816 6556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:12.541312 6556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:12.541586 6556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:12.559132 6556 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44207
I20260924 00:50:12.559166 6622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44207 every 8 connection(s)
I20260924 00:50:12.560838 6556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:12.565249 6623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:12.567790 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 6556
I20260924 00:50:12.568053 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:12.568902 888 external_mini_cluster.cc:1428] Setting key 9e2870dea91297842421e6c857683a97
I20260924 00:50:12.579856 6623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af: Bootstrap starting.
I20260924 00:50:12.584084 6623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:12.585541 6623 log.cc:824] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af: Log is configured to *not* fsync() on all Append() calls
Sep 24 00:50:12 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:12.588882 6623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af: No bootstrap required, opened a new log
I20260924 00:50:12.594936 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:12.571441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.0.222.62:44207)
0924 00:50:12.572523 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:12.572542 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:12.573064 (+ 522us) server_negotiation.cc:407] Connection header received
0924 00:50:12.575042 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:12.575069 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:12.575568 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:12.575973 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:12.578031 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:12.579408 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:12.581341 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:12.582056 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:12.582192 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:12.587286 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:12.587295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:12.587310 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:12.587364 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:12.590350 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:12.591227 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:12.591235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:12.591243 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:12.591567 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:12.591977 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:12.591982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:12.591986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:12.592563 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:12.592855 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:12.593351 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 00:50:12.593860 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":452,"threads_started":1}
I20260924 00:50:12.597888 6623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad73d3d4c964d38a3100dfa2a89c0af" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44207 } }
I20260924 00:50:12.598332 6623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:12.598424 6623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad73d3d4c964d38a3100dfa2a89c0af, State: Initialized, Role: FOLLOWER
I20260924 00:50:12.598992 6623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad73d3d4c964d38a3100dfa2a89c0af" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44207 } }
I20260924 00:50:12.599216 6623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:12.599342 6623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:12.599505 6623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:12.601915 6623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad73d3d4c964d38a3100dfa2a89c0af" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44207 } }
I20260924 00:50:12.602541 6623 leader_election.cc:302] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ad73d3d4c964d38a3100dfa2a89c0af; no voters:
I20260924 00:50:12.603173 6623 leader_election.cc:288] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:12.603299 6628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:12.604012 6628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [term 1 LEADER]: Becoming Leader. State: Replica: 9ad73d3d4c964d38a3100dfa2a89c0af, State: Running, Role: LEADER
I20260924 00:50:12.604547 6628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad73d3d4c964d38a3100dfa2a89c0af" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44207 } }
I20260924 00:50:12.605268 6623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:12.607878 6629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ad73d3d4c964d38a3100dfa2a89c0af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad73d3d4c964d38a3100dfa2a89c0af" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44207 } } }
I20260924 00:50:12.608425 6630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ad73d3d4c964d38a3100dfa2a89c0af. Latest consensus state: current_term: 1 leader_uuid: "9ad73d3d4c964d38a3100dfa2a89c0af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad73d3d4c964d38a3100dfa2a89c0af" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44207 } } }
I20260924 00:50:12.608723 6629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:12.608847 6630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:12.610076 6636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:12.614926 6636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:12.623946 6636 catalog_manager.cc:1347] Generated new cluster ID: b67dbcd2a9604abd85bf1c2ad5da649d
I20260924 00:50:12.624080 6636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:12.634517 6636 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:12.636536 6636 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:12.648957 6636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ad73d3d4c964d38a3100dfa2a89c0af: Generated new TSK 0
I20260924 00:50:12.650175 6636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:50:12.710161 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44207
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:13.043154 6649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:13.043440 6649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:13.043560 6649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:13.051148 6649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:13.051271 6649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:13.051411 6649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:13.061978 6649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:13.063371 6649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:13.064790 6649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:13.079146 6654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:13.079178 6655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:13.079083 6657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:13.080319 6649 server_base.cc:1034] running on GCE node
I20260924 00:50:13.081102 6649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:13.082170 6649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:13.083412 6649 hybrid_clock.cc:648] HybridClock initialized: now 1790211013083353 us; error 47 us; skew 500 ppm
I20260924 00:50:13.083768 6649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:13 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:13.087064 6649 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:50:13.088653 6649 webserver.cc:460] Webserver started at http://127.0.222.1:38217/ using document root <none> and password file <none>
I20260924 00:50:13.089457 6649 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:13.089561 6649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:13.089820 6649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:13.092851 6649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "d20d660972e5463a83d4bf2618088def"
format_stamp: "Formatted at 2026-09-24 00:50:13 on dist-test-slave-rtvz"
server_key: "40a55cf1a70b8f752091686eca29acf1"
server_key_iv: "0d0ded50181f14f5d184d0481b70af6e"
server_key_version: "encryptionkey@0"
I20260924 00:50:13.093675 6649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "d20d660972e5463a83d4bf2618088def"
format_stamp: "Formatted at 2026-09-24 00:50:13 on dist-test-slave-rtvz"
server_key: "40a55cf1a70b8f752091686eca29acf1"
server_key_iv: "0d0ded50181f14f5d184d0481b70af6e"
server_key_version: "encryptionkey@0"
I20260924 00:50:13.099339 6649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:50:13.103497 6664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:13.105125 6649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:13.105388 6649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "d20d660972e5463a83d4bf2618088def"
format_stamp: "Formatted at 2026-09-24 00:50:13 on dist-test-slave-rtvz"
server_key: "40a55cf1a70b8f752091686eca29acf1"
server_key_iv: "0d0ded50181f14f5d184d0481b70af6e"
server_key_version: "encryptionkey@0"
I20260924 00:50:13.105600 6649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:13.124965 6649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:13.129427 6649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:13.129702 6649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:13.130744 6649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:13.132274 6649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:13.132375 6649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:13.132494 6649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:13.132546 6649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:13.163626 6649 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:41627
I20260924 00:50:13.163668 6777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:41627 every 8 connection(s)
I20260924 00:50:13.164994 6649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:13.172623 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 6649
I20260924 00:50:13.172842 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:50:13.173532 888 external_mini_cluster.cc:1428] Setting key 6a8f76db8d21a55f0abb4244e00386db
Sep 24 00:50:13 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:13.185598 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:13.167592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:49459 (local address 127.0.222.62:44207)
0924 00:50:13.168089 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:13.168093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:13.169147 (+ 1054us) server_negotiation.cc:407] Connection header received
0924 00:50:13.171033 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:13.171038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:13.171130 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:13.171229 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:13.172907 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:13.173732 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:13.175878 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:13.176304 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:13.176424 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:13.179898 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:13.179904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:13.179907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:13.179941 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:13.182426 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:13.183189 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:13.183193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:13.183195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:13.183323 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:13.183750 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:13.183754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:13.183756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:13.184043 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:13.184176 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:13.184944 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 00:50:13.185189 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":275,"threads_started":1}
I20260924 00:50:13.186628 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:13.168200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44207 (local address 127.0.222.1:49459)
0924 00:50:13.168977 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:13.169020 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:13.170729 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:13.171495 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:13.171504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:13.172054 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:13.172705 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:13.172732 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:13.173892 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:13.173896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:13.175738 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:13.175752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:13.176534 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:13.176541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:13.176777 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:13.177452 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:13.177481 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:13.179658 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:13.182572 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:13.182581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:13.182600 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:13.183063 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:13.183430 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:13.183436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:13.183440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:13.183642 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:13.184172 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:13.184181 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:13.184554 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 00:50:13.184818 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 00:50:13.185458 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":294,"threads_started":1}
I20260924 00:50:13.188277 6778 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44207
I20260924 00:50:13.188594 6778 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:13.189543 6778 heartbeater.cc:511] Master 127.0.222.62:44207 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:50:13.191910 6588 ts_manager.cc:194] Registered new tserver with Master: d20d660972e5463a83d4bf2618088def (127.0.222.1:41627)
I20260924 00:50:13.193567 6588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:49459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:50:13.216625 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44207
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:13.536746 6784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:13.537043 6784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:13.537168 6784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:13.544870 6784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:13.544997 6784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:13.545146 6784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:13.555347 6784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:13.556671 6784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:13.557998 6784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:13.573676 6789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:13.573896 6784 server_base.cc:1034] running on GCE node
W20260924 00:50:13.573650 6790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:13.574203 6792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:13.574915 6784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:13.575920 6784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:13.577128 6784 hybrid_clock.cc:648] HybridClock initialized: now 1790211013577078 us; error 44 us; skew 500 ppm
I20260924 00:50:13.577457 6784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:13 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:13.580525 6784 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:50:13.582075 6784 webserver.cc:460] Webserver started at http://127.0.222.2:38165/ using document root <none> and password file <none>
I20260924 00:50:13.582757 6784 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:13.582877 6784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:13.583161 6784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:13.586413 6784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "cff1df8c3eaa4a80b5b8ee609d9f4868"
format_stamp: "Formatted at 2026-09-24 00:50:13 on dist-test-slave-rtvz"
server_key: "8842346b3d2ac7c79b52a0ab43535232"
server_key_iv: "3a601e301750bc7a04fd877d074dc545"
server_key_version: "encryptionkey@0"
I20260924 00:50:13.587239 6784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "cff1df8c3eaa4a80b5b8ee609d9f4868"
format_stamp: "Formatted at 2026-09-24 00:50:13 on dist-test-slave-rtvz"
server_key: "8842346b3d2ac7c79b52a0ab43535232"
server_key_iv: "3a601e301750bc7a04fd877d074dc545"
server_key_version: "encryptionkey@0"
I20260924 00:50:13.592860 6784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:50:13.596856 6799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:13.598443 6784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:13.598619 6784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "cff1df8c3eaa4a80b5b8ee609d9f4868"
format_stamp: "Formatted at 2026-09-24 00:50:13 on dist-test-slave-rtvz"
server_key: "8842346b3d2ac7c79b52a0ab43535232"
server_key_iv: "3a601e301750bc7a04fd877d074dc545"
server_key_version: "encryptionkey@0"
I20260924 00:50:13.598822 6784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:13.613633 6784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:13.618335 6784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:13.618621 6784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:13.619642 6784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:13.621065 6784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:13.621151 6784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:13.621289 6784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:13.621356 6784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:13.652001 6784 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:35231
I20260924 00:50:13.652041 6912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:35231 every 8 connection(s)
I20260924 00:50:13.653386 6784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:13.656252 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 6784
I20260924 00:50:13.656445 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:50:13.656842 888 external_mini_cluster.cc:1428] Setting key a2681e411700ededb1788a8169797818
Sep 24 00:50:13 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:50:13.674437 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:13.656086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45229 (local address 127.0.222.62:44207)
0924 00:50:13.656296 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:13.656302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:13.657667 (+ 1365us) server_negotiation.cc:407] Connection header received
0924 00:50:13.659547 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:13.659553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:13.659632 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:13.659733 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:13.661688 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:13.662651 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:13.664463 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:13.664894 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:13.665012 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:13.668633 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:13.668639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:13.668643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:13.668681 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:13.671167 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:13.671971 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:13.671975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:13.671977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:13.672113 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:13.672553 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:13.672557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:13.672559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:13.672836 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:13.672965 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:13.673827 (+ 862us) server_negotiation.cc:299] Negotiation successful
0924 00:50:13.674085 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 00:50:13.675527 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:13.656604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44207 (local address 127.0.222.2:45229)
0924 00:50:13.657492 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:13.657551 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:13.659246 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:13.660078 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:13.660088 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:13.660702 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:13.661502 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:13.661524 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:13.662830 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:13.662834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:13.664276 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:13.664295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:13.665134 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:13.665141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:13.665422 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:13.666092 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:13.666117 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:13.668368 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:13.671347 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:13.671358 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:13.671380 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:13.671840 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:13.672222 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:13.672227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:13.672231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:13.672437 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:13.672958 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:13.672967 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:13.673393 (+ 426us) client_negotiation.cc:769] Sending connection context
0924 00:50:13.673717 (+ 324us) client_negotiation.cc:240] Negotiation successful
0924 00:50:13.674309 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":334,"threads_started":1}
I20260924 00:50:13.677021 6913 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:50:13.677349 6913 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:13.678097 6913 heartbeater.cc:511] Master 127.0.222.62:44207 requested a full tablet report, sending...
I20260924 00:50:13.679622 6588 ts_manager.cc:194] Registered new tserver with Master: cff1df8c3eaa4a80b5b8ee609d9f4868 (127.0.222.2:35231)
I20260924 00:50:13.680480 6588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:45229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:50:13.701449 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44207
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:14.031455 6918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:14.031778 6918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:14.031890 6918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:14.039263 6918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:14.039398 6918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:14.039515 6918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:14.049598 6918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:14.050951 6918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:14.052266 6918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:14.065949 6923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:14.065903 6926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:14.065925 6924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:14.066707 6918 server_base.cc:1034] running on GCE node
I20260924 00:50:14.067368 6918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:14.068411 6918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:14.069643 6918 hybrid_clock.cc:648] HybridClock initialized: now 1790211014069573 us; error 49 us; skew 500 ppm
I20260924 00:50:14.070096 6918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:14.073742 6918 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:50:14.075528 6918 webserver.cc:460] Webserver started at http://127.0.222.3:40937/ using document root <none> and password file <none>
I20260924 00:50:14.076295 6918 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:14.076433 6918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:14.076748 6918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:14.080052 6918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "869250218a2a42d88892aceae14c093b"
format_stamp: "Formatted at 2026-09-24 00:50:14 on dist-test-slave-rtvz"
server_key: "a2b2589a4197d68e7099303e9b441098"
server_key_iv: "cd8e0442f525711808548418936ad68c"
server_key_version: "encryptionkey@0"
I20260924 00:50:14.080938 6918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "869250218a2a42d88892aceae14c093b"
format_stamp: "Formatted at 2026-09-24 00:50:14 on dist-test-slave-rtvz"
server_key: "a2b2589a4197d68e7099303e9b441098"
server_key_iv: "cd8e0442f525711808548418936ad68c"
server_key_version: "encryptionkey@0"
I20260924 00:50:14.086961 6918 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 00:50:14.091032 6933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:14.092621 6918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:50:14.092818 6918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "869250218a2a42d88892aceae14c093b"
format_stamp: "Formatted at 2026-09-24 00:50:14 on dist-test-slave-rtvz"
server_key: "a2b2589a4197d68e7099303e9b441098"
server_key_iv: "cd8e0442f525711808548418936ad68c"
server_key_version: "encryptionkey@0"
I20260924 00:50:14.093042 6918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:14.117285 6918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:14.121461 6918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:14.121717 6918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:14.122682 6918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:14.123997 6918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:14.124075 6918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:14.124156 6918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:14.124198 6918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:14.154824 6918 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:38611
I20260924 00:50:14.154930 7046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:38611 every 8 connection(s)
I20260924 00:50:14.156193 6918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:14.161875 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 6918
I20260924 00:50:14.162098 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:50:14.162693 888 external_mini_cluster.cc:1428] Setting key 889872b06bbdfca45ab31a14b16e3ab2
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:14.178895 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:14.158964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:46723 (local address 127.0.222.62:44207)
0924 00:50:14.159151 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.159156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.160423 (+ 1267us) server_negotiation.cc:407] Connection header received
0924 00:50:14.162801 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.162807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.162930 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.163039 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.165426 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.166611 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.168347 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.168935 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.169048 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:14.172669 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:14.172675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:14.172678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:14.172719 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:14.175321 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.176241 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.176245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.176248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.176385 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.176838 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.176842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.176844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.177238 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:14.177380 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:14.178280 (+ 900us) server_negotiation.cc:299] Negotiation successful
0924 00:50:14.178515 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:50:14.179847 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:14.159394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44207 (local address 127.0.222.3:46723)
0924 00:50:14.160234 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:14.160280 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:14.162444 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:14.163339 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:14.163349 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:14.164134 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:14.165137 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:14.165167 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.166769 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.166772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:14.168184 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:14.168199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.169218 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.169228 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:14.169462 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:14.170172 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:14.170198 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:14.172421 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:14.175487 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:14.175495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:14.175510 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:14.176079 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:14.176503 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:14.176508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:14.176511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:14.176716 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:14.177363 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:14.177372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:14.177823 (+ 451us) client_negotiation.cc:769] Sending connection context
0924 00:50:14.178104 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 00:50:14.178632 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260924 00:50:14.181425 7047 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44207
I20260924 00:50:14.181748 7047 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:14.182548 7047 heartbeater.cc:511] Master 127.0.222.62:44207 requested a full tablet report, sending...
I20260924 00:50:14.184228 6588 ts_manager.cc:194] Registered new tserver with Master: 869250218a2a42d88892aceae14c093b (127.0.222.3:38611)
I20260924 00:50:14.185070 6588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:46723
I20260924 00:50:14.191848 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:50:14.197016 6778 heartbeater.cc:503] Master 127.0.222.62:44207 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/test-admin.keytab.
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:14.219125 888 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 00:50:14.224551 7054 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:14.221839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.0.222.1:41627)
0924 00:50:14.222602 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.222609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.222639 (+ 30us) server_negotiation.cc:407] Connection header received
0924 00:50:14.222720 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.222727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.222954 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.223099 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.224085 (+ 986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46422: BlockingRecv error: recv got EOF from 127.0.0.1:46422 (error 108)
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":473,"threads_started":1}
W20260924 00:50:14.224762 888 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:14.226608 888 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:14.228022 888 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:50:14.244943 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:14.229900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.0.222.1:41627)
0924 00:50:14.230101 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.230108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.230440 (+ 332us) server_negotiation.cc:407] Connection header received
0924 00:50:14.230630 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.230636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.230705 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.230786 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.231856 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.232858 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.234202 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.234830 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.234962 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:14.238966 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:14.238982 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:14.238999 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:14.239044 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:14.242148 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.242883 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.242890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.242893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.243023 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.243378 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.243382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.243384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.243741 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:14.243974 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:14.244377 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 00:50:14.244652 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 00:50:14.250830 7054 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:14.249257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.0.222.1:41627)
0924 00:50:14.249478 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.249484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.249632 (+ 148us) server_negotiation.cc:407] Connection header received
0924 00:50:14.249776 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.249782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.249840 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.249915 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.250595 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46444: BlockingRecv error: recv got EOF from 127.0.0.1:46444 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260924 00:50:14.251071 888 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:14.252547 888 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:14.253832 888 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:50:14.268014 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:14.255073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.222.1:41627)
0924 00:50:14.255234 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.255240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.255520 (+ 280us) server_negotiation.cc:407] Connection header received
0924 00:50:14.255664 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.255668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.255727 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.255804 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.256741 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.257510 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.258703 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.259104 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.259223 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:14.262895 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:14.262900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:14.262904 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:14.262939 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:14.265479 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.266220 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.266224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.266227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.266389 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.266742 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.266746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.266748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.267025 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:14.267265 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:14.267549 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 00:50:14.267750 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 00:50:14.272178 7054 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:14.270713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.222.1:41627)
0924 00:50:14.270913 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.270919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.271059 (+ 140us) server_negotiation.cc:407] Connection header received
0924 00:50:14.271189 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.271192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.271261 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.271337 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.271977 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46464: BlockingRecv error: recv got EOF from 127.0.0.1:46464 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 00:50:14.272334 888 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790210962623352-888-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:14.273904 888 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:14.275287 888 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:50:14.290223 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:14.276600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.222.1:41627)
0924 00:50:14.276753 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.276799 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.277161 (+ 362us) server_negotiation.cc:407] Connection header received
0924 00:50:14.277427 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.277432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.277503 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.277589 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.278605 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.279413 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.280661 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.281148 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.281284 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:14.284853 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:14.284858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:14.284861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:14.284898 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:14.287610 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.288315 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.288319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.288321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.288466 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.288814 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.288818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.288820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.289091 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:14.289400 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:14.289659 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 00:50:14.289878 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 00:50:14.291692 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 6649
I20260924 00:50:14.304348 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:14.292379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.0.222.1:41627)
0924 00:50:14.292521 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.292526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.292900 (+ 374us) server_negotiation.cc:407] Connection header received
0924 00:50:14.293059 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.293064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.293128 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.293255 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.294279 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.294947 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.296156 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.296556 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.296688 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:14.299008 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:14.299013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:14.299016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:14.299050 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:14.301540 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.302206 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.302211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.302213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.302357 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.302709 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.302713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.302715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.303064 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:14.303351 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:14.303679 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 00:50:14.303912 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:50:14.304999 6773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:14.389532 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 6649
I20260924 00:50:14.402681 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 6784
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:50:14.419725 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:14.403622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.0.222.2:35231)
0924 00:50:14.404162 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.404172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.404197 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:50:14.404284 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.404291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.404545 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.404675 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.405764 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.406935 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.408315 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.408893 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.409005 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:14.413141 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:14.413157 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:14.413175 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:14.413248 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:14.416525 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.417607 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.417616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.417618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.417764 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.418213 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.418216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.418218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.418574 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:14.418826 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:14.419214 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 00:50:14.419436 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":287,"threads_started":1}
I20260924 00:50:14.421041 6908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:14.504026 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 6784
I20260924 00:50:14.516820 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 6918
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:50:14.533028 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:14.517827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.0.222.3:38611)
0924 00:50:14.518435 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:14.518442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:14.518470 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:50:14.518529 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:14.518536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:14.518744 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:14.518901 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:14.520007 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.520997 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.522225 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:14.522770 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:14.522890 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:14.527085 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:14.527098 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:14.527113 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:14.527153 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:14.530172 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.530988 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.530995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.530998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.531128 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:14.531516 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:14.531520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:14.531522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:14.531897 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:14.532135 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:14.532401 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 00:50:14.532630 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":297,"threads_started":1}
I20260924 00:50:14.534400 7042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:14.616242 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 6918
I20260924 00:50:14.628551 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 6556
I20260924 00:50:14.629822 6618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:14.733631 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 6556
2026-09-24T00:50:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3819 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[7074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[7074](info): set up 2 sockets
Sep 24 00:50:14 dist-test-slave-rtvz krb5kdc[7074](info): commencing operation
krb5kdc: starting...
W20260924 00:50:15.752895 888 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:50:15 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:50:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.62.keytab.
Entry for principal oryx/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.62.keytab.
I20260924 00:50:15.961316 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42943
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.62.keytab
--principal=oryx/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:16.289716 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:16.290030 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:16.290158 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:16.297489 7087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:16.297614 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:16.297691 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:16.297767 7087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:16.297827 7087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:16.307928 7087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.62.keytab
--principal=oryx/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:16.309355 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:16.310684 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:16.322731 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:16.322737 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:16.322781 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:16.323596 7087 server_base.cc:1034] running on GCE node
I20260924 00:50:16.324370 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:16.325933 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:16.327136 7087 hybrid_clock.cc:648] HybridClock initialized: now 1790211016327082 us; error 48 us; skew 500 ppm
I20260924 00:50:16.327487 7087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:16 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211016, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:16.330691 7087 init.cc:377] Logged in from keytab as oryx/127.0.222.62@KRBTEST.COM (short username oryx)
I20260924 00:50:16.332223 7087 webserver.cc:460] Webserver started at http://127.0.222.62:46839/ using document root <none> and password file <none>
I20260924 00:50:16.332924 7087 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:16.333026 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:16.333331 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:16.336572 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "1bff8d21582e43049659eaed39fe3937"
format_stamp: "Formatted at 2026-09-24 00:50:16 on dist-test-slave-rtvz"
server_key: "7cd27831c5e23331fc448c75f02288cd"
server_key_iv: "f62e6769e6ec998beedee9f49470e5e8"
server_key_version: "encryptionkey@0"
I20260924 00:50:16.337414 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "1bff8d21582e43049659eaed39fe3937"
format_stamp: "Formatted at 2026-09-24 00:50:16 on dist-test-slave-rtvz"
server_key: "7cd27831c5e23331fc448c75f02288cd"
server_key_iv: "f62e6769e6ec998beedee9f49470e5e8"
server_key_version: "encryptionkey@0"
I20260924 00:50:16.343575 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 00:50:16.347661 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:16.349375 7087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:16.349557 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "1bff8d21582e43049659eaed39fe3937"
format_stamp: "Formatted at 2026-09-24 00:50:16 on dist-test-slave-rtvz"
server_key: "7cd27831c5e23331fc448c75f02288cd"
server_key_iv: "f62e6769e6ec998beedee9f49470e5e8"
server_key_version: "encryptionkey@0"
I20260924 00:50:16.349762 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:16.378899 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:16.383458 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:16.383718 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:16.402107 7087 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42943
I20260924 00:50:16.402128 7154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42943 every 8 connection(s)
I20260924 00:50:16.403563 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:16.407565 7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:16.412324 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 7087
I20260924 00:50:16.412587 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:16.413384 888 external_mini_cluster.cc:1428] Setting key 56f8521befc8191bd66ea65fda08a2e7
I20260924 00:50:16.420974 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: Bootstrap starting.
I20260924 00:50:16.424963 7155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:16.426393 7155 log.cc:824] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: Log is configured to *not* fsync() on all Append() calls
Sep 24 00:50:16 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:16.430074 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: No bootstrap required, opened a new log
I20260924 00:50:16.439750 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:16.415626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.0.222.62:42943)
0924 00:50:16.416559 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:16.416576 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:16.416634 (+ 58us) server_negotiation.cc:407] Connection header received
0924 00:50:16.419169 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:16.419207 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:16.419831 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:16.420347 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:16.421949 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:16.424004 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:16.425500 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:16.426623 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:16.426935 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:16.431270 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:16.431279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:16.431297 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:16.431353 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:16.434904 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:16.435947 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:16.435955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:16.435963 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:16.436191 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:16.436636 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:16.436639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:16.436641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:16.437387 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:16.437640 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:16.438172 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 00:50:16.438621 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":396,"threads_started":1}
I20260924 00:50:16.439653 7155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } }
I20260924 00:50:16.440085 7155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:16.440168 7155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bff8d21582e43049659eaed39fe3937, State: Initialized, Role: FOLLOWER
I20260924 00:50:16.440750 7155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } }
I20260924 00:50:16.440949 7155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:16.441040 7155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:16.441219 7155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:16.443714 7155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } }
I20260924 00:50:16.444255 7155 leader_election.cc:302] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bff8d21582e43049659eaed39fe3937; no voters:
I20260924 00:50:16.444832 7155 leader_election.cc:288] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:16.444960 7160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:16.446048 7160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 1 LEADER]: Becoming Leader. State: Replica: 1bff8d21582e43049659eaed39fe3937, State: Running, Role: LEADER
I20260924 00:50:16.446583 7160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } }
I20260924 00:50:16.447114 7155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:16.449323 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bff8d21582e43049659eaed39fe3937. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } }
I20260924 00:50:16.449295 7161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } }
I20260924 00:50:16.449713 7161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:16.449713 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:16.451905 7169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:16.457490 7169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:16.467314 7169 catalog_manager.cc:1347] Generated new cluster ID: 7981b32f7ae2469a905d7dc221204823
I20260924 00:50:16.467454 7169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:16.484416 7169 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:16.485950 7169 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:16.496640 7169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: Generated new TSK 0
I20260924 00:50:16.497905 7169 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab.
I20260924 00:50:16.554222 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42943
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:16.892710 7182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:16.893011 7182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:16.893132 7182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:16.900799 7182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:16.900928 7182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:16.901079 7182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:16.911237 7182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:16.912603 7182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:16.914047 7182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:16.927556 7187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:16.927490 7188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:16.927533 7190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:16.930619 7182 server_base.cc:1034] running on GCE node
I20260924 00:50:16.931267 7182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:16.932459 7182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:16.933701 7182 hybrid_clock.cc:648] HybridClock initialized: now 1790211016933653 us; error 35 us; skew 500 ppm
I20260924 00:50:16.934077 7182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:16 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211016, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:16.937331 7182 init.cc:377] Logged in from keytab as oryx/127.0.222.1@KRBTEST.COM (short username oryx)
I20260924 00:50:16.938939 7182 webserver.cc:460] Webserver started at http://127.0.222.1:46707/ using document root <none> and password file <none>
I20260924 00:50:16.939666 7182 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:16.939771 7182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:16.940052 7182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:16.943161 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "3359189faab547c5a64c98e926d18930"
format_stamp: "Formatted at 2026-09-24 00:50:16 on dist-test-slave-rtvz"
server_key: "ea750cf156f9961486d86b30d4a76ffc"
server_key_iv: "ede9c606637d09d79ce6a333a4c2b5d3"
server_key_version: "encryptionkey@0"
I20260924 00:50:16.943986 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "3359189faab547c5a64c98e926d18930"
format_stamp: "Formatted at 2026-09-24 00:50:16 on dist-test-slave-rtvz"
server_key: "ea750cf156f9961486d86b30d4a76ffc"
server_key_iv: "ede9c606637d09d79ce6a333a4c2b5d3"
server_key_version: "encryptionkey@0"
I20260924 00:50:16.949717 7182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 00:50:16.953760 7197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:16.955241 7182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:50:16.955421 7182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "3359189faab547c5a64c98e926d18930"
format_stamp: "Formatted at 2026-09-24 00:50:16 on dist-test-slave-rtvz"
server_key: "ea750cf156f9961486d86b30d4a76ffc"
server_key_iv: "ede9c606637d09d79ce6a333a4c2b5d3"
server_key_version: "encryptionkey@0"
I20260924 00:50:16.955633 7182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:16.982898 7182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:16.987751 7182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:16.988055 7182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:16.989146 7182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:16.990619 7182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:16.990694 7182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:16.990795 7182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:16.990857 7182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:17.022048 7182 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:46755
I20260924 00:50:17.022107 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:46755 every 8 connection(s)
I20260924 00:50:17.023361 7182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:17.025162 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 7182
I20260924 00:50:17.025465 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:50:17.026139 888 external_mini_cluster.cc:1428] Setting key c05f26db7cd3bc3eacf2411afe8d45d6
Sep 24 00:50:17 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211016, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:50:17.048394 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:17.026282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55941 (local address 127.0.222.62:42943)
0924 00:50:17.026768 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:17.026775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:17.027833 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 00:50:17.030459 (+ 2626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:17.030463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:17.030533 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:17.030630 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:17.032698 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:17.033547 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:17.035913 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:17.036550 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:17.036694 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:17.040932 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:17.040939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:17.040944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:17.040992 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:17.044342 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:17.045341 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:17.045349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:17.045353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:17.045541 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:17.046206 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:17.046211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:17.046215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:17.046594 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:17.046771 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:17.047700 (+ 929us) server_negotiation.cc:299] Negotiation successful
0924 00:50:17.047997 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
I20260924 00:50:17.049450 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:17.026918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:55941)
0924 00:50:17.027664 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:17.027709 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:17.030110 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:17.030929 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:17.030942 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:17.031667 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:17.032504 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:17.032530 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:17.033725 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:17.033731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:17.035676 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:17.035697 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:17.036845 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:17.036852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:17.037069 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:17.037913 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:17.037939 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:17.040604 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:17.044574 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:17.044583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:17.044600 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:17.045141 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:17.045724 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:17.045728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:17.045731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:17.046063 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:17.046735 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:17.046822 (+ 87us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:17.047244 (+ 422us) client_negotiation.cc:769] Sending connection context
0924 00:50:17.047538 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 00:50:17.048077 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":306,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.2@KRBTEST.COM" created.
I20260924 00:50:17.051452 7311 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42943
I20260924 00:50:17.051885 7311 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:17.052953 7311 heartbeater.cc:511] Master 127.0.222.62:42943 requested a full tablet report, sending...
I20260924 00:50:17.055948 7119 ts_manager.cc:194] Registered new tserver with Master: 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755)
I20260924 00:50:17.057674 7119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:55941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab.
I20260924 00:50:17.077322 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42943
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:17.401734 7318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:17.402045 7318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:17.402174 7318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:17.409749 7318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:17.409878 7318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:17.410048 7318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:17.420243 7318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:17.421676 7318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:17.423012 7318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:17.436321 7323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:17.436401 7326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:17.436405 7324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:17.437026 7318 server_base.cc:1034] running on GCE node
I20260924 00:50:17.437727 7318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:17.438704 7318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:17.440001 7318 hybrid_clock.cc:648] HybridClock initialized: now 1790211017439954 us; error 31 us; skew 500 ppm
I20260924 00:50:17.440388 7318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:17 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211017, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:17.443639 7318 init.cc:377] Logged in from keytab as oryx/127.0.222.2@KRBTEST.COM (short username oryx)
I20260924 00:50:17.445474 7318 webserver.cc:460] Webserver started at http://127.0.222.2:42051/ using document root <none> and password file <none>
I20260924 00:50:17.446273 7318 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:17.446388 7318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:17.446712 7318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:17.449990 7318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "e5ddd2e4fe40404b903b5adeda9646ae"
format_stamp: "Formatted at 2026-09-24 00:50:17 on dist-test-slave-rtvz"
server_key: "d6be69bc89ae9c06dbb037dcdbb6992d"
server_key_iv: "4198a849251d289f90e41f59e86856dd"
server_key_version: "encryptionkey@0"
I20260924 00:50:17.450811 7318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "e5ddd2e4fe40404b903b5adeda9646ae"
format_stamp: "Formatted at 2026-09-24 00:50:17 on dist-test-slave-rtvz"
server_key: "d6be69bc89ae9c06dbb037dcdbb6992d"
server_key_iv: "4198a849251d289f90e41f59e86856dd"
server_key_version: "encryptionkey@0"
I20260924 00:50:17.456274 7318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 00:50:17.460009 7333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:17.461490 7318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:17.461668 7318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "e5ddd2e4fe40404b903b5adeda9646ae"
format_stamp: "Formatted at 2026-09-24 00:50:17 on dist-test-slave-rtvz"
server_key: "d6be69bc89ae9c06dbb037dcdbb6992d"
server_key_iv: "4198a849251d289f90e41f59e86856dd"
server_key_version: "encryptionkey@0"
I20260924 00:50:17.461917 7318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:17.478813 7318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:17.483163 7318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:17.483412 7318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:17.484372 7318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:17.485826 7318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:17.485908 7318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:17.485994 7318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:17.486035 7318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:17.515867 7318 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:40137
I20260924 00:50:17.515911 7445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:40137 every 8 connection(s)
I20260924 00:50:17.517094 7318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:17.527393 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 7318
I20260924 00:50:17.527586 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:50:17.528090 888 external_mini_cluster.cc:1428] Setting key fc944396a384b62cf19a1df6f19cb307
Sep 24 00:50:17 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211017, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:17.541930 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:17.519630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:44339 (local address 127.0.222.62:42943)
0924 00:50:17.519794 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:17.519799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:17.520934 (+ 1135us) server_negotiation.cc:407] Connection header received
0924 00:50:17.522805 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:17.522811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:17.522901 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:17.522991 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:17.524508 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:17.525330 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:17.526838 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:17.527246 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:17.527358 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:17.535933 (+ 8575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:17.535939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:17.535942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:17.535980 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:17.538666 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:17.539492 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:17.539497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:17.539500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:17.539660 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:17.540113 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:17.540116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:17.540118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:17.540408 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:17.540544 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:17.541331 (+ 787us) server_negotiation.cc:299] Negotiation successful
0924 00:50:17.541588 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 00:50:17.542969 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:17.520085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:44339)
0924 00:50:17.520772 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:17.520810 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:17.522475 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:17.523231 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:17.523240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:17.523760 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:17.524318 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:17.524336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:17.525472 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:17.525476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:17.526697 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:17.526708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:17.527480 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:17.527488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:17.527741 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:17.533061 (+ 5320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:17.533093 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:17.535647 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:17.538848 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:17.538859 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:17.538878 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:17.539356 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:17.539782 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:17.539787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:17.539791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:17.539996 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:17.540538 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:17.540547 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:17.540936 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 00:50:17.541213 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 00:50:17.541799 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260924 00:50:17.544682 7446 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42943
I20260924 00:50:17.544993 7446 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:17.545778 7446 heartbeater.cc:511] Master 127.0.222.62:42943 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:50:17.547410 7119 ts_manager.cc:194] Registered new tserver with Master: e5ddd2e4fe40404b903b5adeda9646ae (127.0.222.2:40137)
I20260924 00:50:17.548254 7119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:44339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab.
I20260924 00:50:17.573185 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42943
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:17.895962 7451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:17.896279 7451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:17.896409 7451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:17.903788 7451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:17.903962 7451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:17.904125 7451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:17.914179 7451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:17.915541 7451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:17.916886 7451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:17.930509 7457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:17.930544 7458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:17.930482 7460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:17.931200 7451 server_base.cc:1034] running on GCE node
I20260924 00:50:17.931888 7451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:17.933043 7451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:17.934283 7451 hybrid_clock.cc:648] HybridClock initialized: now 1790211017934218 us; error 68 us; skew 500 ppm
I20260924 00:50:17.934629 7451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:17 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211017, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:17.938086 7451 init.cc:377] Logged in from keytab as oryx/127.0.222.3@KRBTEST.COM (short username oryx)
I20260924 00:50:17.939639 7451 webserver.cc:460] Webserver started at http://127.0.222.3:46817/ using document root <none> and password file <none>
I20260924 00:50:17.940354 7451 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:17.940476 7451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:17.940770 7451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:17.943857 7451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "c1d92fed7ae44e08b7a54f52c9c03978"
format_stamp: "Formatted at 2026-09-24 00:50:17 on dist-test-slave-rtvz"
server_key: "4bb8142c256abebd04f50d382f1698a0"
server_key_iv: "cd4336dd07e1b5f72d6929997496ab8f"
server_key_version: "encryptionkey@0"
I20260924 00:50:17.944662 7451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "c1d92fed7ae44e08b7a54f52c9c03978"
format_stamp: "Formatted at 2026-09-24 00:50:17 on dist-test-slave-rtvz"
server_key: "4bb8142c256abebd04f50d382f1698a0"
server_key_iv: "cd4336dd07e1b5f72d6929997496ab8f"
server_key_version: "encryptionkey@0"
I20260924 00:50:17.950294 7451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 00:50:17.954317 7467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:17.955987 7451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:50:17.956179 7451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "c1d92fed7ae44e08b7a54f52c9c03978"
format_stamp: "Formatted at 2026-09-24 00:50:17 on dist-test-slave-rtvz"
server_key: "4bb8142c256abebd04f50d382f1698a0"
server_key_iv: "cd4336dd07e1b5f72d6929997496ab8f"
server_key_version: "encryptionkey@0"
I20260924 00:50:17.956403 7451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:17.976351 7451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:17.980494 7451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:17.980770 7451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:17.981827 7451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:17.983212 7451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:17.983299 7451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:17.983424 7451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:17.983484 7451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:18.014997 7451 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33573
I20260924 00:50:18.015058 7579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33573 every 8 connection(s)
I20260924 00:50:18.016453 7451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:18.023252 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 7451
I20260924 00:50:18.023454 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:50:18.023880 888 external_mini_cluster.cc:1428] Setting key 61923e060f4094972edf2712053cb28a
Sep 24 00:50:18 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211017, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:18.037103 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.019093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:44149 (local address 127.0.222.62:42943)
0924 00:50:18.019260 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:18.019264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:18.020430 (+ 1166us) server_negotiation.cc:407] Connection header received
0924 00:50:18.022939 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:18.022945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:18.023035 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:18.023151 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:18.025147 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.026191 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.027893 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.028315 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.028423 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.031825 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:18.031832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:18.031837 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:18.031877 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:18.034214 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.034907 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.034913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.034916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.035060 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.035452 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.035456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.035458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.035718 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:18.035851 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:18.036523 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 00:50:18.036750 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:50:18.038434 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.019515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:44149)
0924 00:50:18.020262 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:18.020303 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:18.022584 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:18.023400 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:18.023412 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:18.024164 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:18.024968 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.025001 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.026335 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.026340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.027762 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.027774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.028532 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.028539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.028762 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.029430 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:18.029456 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:18.031617 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:18.034370 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.034379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.034394 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.034796 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.035162 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.035167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.035171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.035349 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.035838 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:18.035845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:18.036177 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 00:50:18.036426 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 00:50:18.036974 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":311,"threads_started":1}
I20260924 00:50:18.039886 7580 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42943
I20260924 00:50:18.040212 7580 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:18.040987 7580 heartbeater.cc:511] Master 127.0.222.62:42943 requested a full tablet report, sending...
I20260924 00:50:18.042656 7118 ts_manager.cc:194] Registered new tserver with Master: c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573)
I20260924 00:50:18.043599 7118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:44149
I20260924 00:50:18.052105 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:50:18.061470 7311 heartbeater.cc:503] Master 127.0.222.62:42943 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.222.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.61.keytab.
Entry for principal oryx/127.0.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.61.keytab.
I20260924 00:50:18.097247 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:36141
--webserver_interface=127.0.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.61.keytab
--principal=oryx/127.0.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42943,127.0.222.61:36141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:18.455909 7585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:18.456216 7585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:18.456350 7585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:18.465188 7585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:18.465459 7585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:18.465606 7585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:18.465775 7585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:18.465884 7585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:18.479445 7585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42943,127.0.222.61:36141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.61.keytab
--principal=oryx/127.0.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:36141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
W20260924 00:50:18.479970 7585 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.222.62:42943,127.0.222.61:36141'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 00:50:18.481130 7585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:18.482805 7585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:18.497176 7590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:18.497612 7593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:18.497654 7591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:18.499033 7585 server_base.cc:1034] running on GCE node
I20260924 00:50:18.499886 7585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:18.501412 7585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:18.502647 7585 hybrid_clock.cc:648] HybridClock initialized: now 1790211018502566 us; error 43 us; skew 500 ppm
I20260924 00:50:18.502995 7585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:18 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211018, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:18.506656 7585 init.cc:377] Logged in from keytab as oryx/127.0.222.61@KRBTEST.COM (short username oryx)
I20260924 00:50:18.508419 7585 webserver.cc:460] Webserver started at http://127.0.222.61:32837/ using document root <none> and password file <none>
I20260924 00:50:18.509253 7585 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:18.509387 7585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:18.509727 7585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:18.513028 7585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data/instance:
uuid: "6e488b41d2c04decbea7acdaa202c91e"
format_stamp: "Formatted at 2026-09-24 00:50:18 on dist-test-slave-rtvz"
server_key: "d8b3cccf9c8d0f4eb0077893cff944a1"
server_key_iv: "a8851c141480076cc9b1e14bbc31cbd1"
server_key_version: "encryptionkey@0"
I20260924 00:50:18.513962 7585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal/instance:
uuid: "6e488b41d2c04decbea7acdaa202c91e"
format_stamp: "Formatted at 2026-09-24 00:50:18 on dist-test-slave-rtvz"
server_key: "d8b3cccf9c8d0f4eb0077893cff944a1"
server_key_iv: "a8851c141480076cc9b1e14bbc31cbd1"
server_key_version: "encryptionkey@0"
I20260924 00:50:18.520134 7585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260924 00:50:18.525039 7601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:18.527765 7585 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 00:50:18.527940 7585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
uuid: "6e488b41d2c04decbea7acdaa202c91e"
format_stamp: "Formatted at 2026-09-24 00:50:18 on dist-test-slave-rtvz"
server_key: "d8b3cccf9c8d0f4eb0077893cff944a1"
server_key_iv: "a8851c141480076cc9b1e14bbc31cbd1"
server_key_version: "encryptionkey@0"
I20260924 00:50:18.528319 7585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:18.551204 7446 heartbeater.cc:503] Master 127.0.222.62:42943 was elected leader, sending a full tablet report...
I20260924 00:50:18.561172 7585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:18.566072 7585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:18.566408 7585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:18.596410 7585 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:36141
I20260924 00:50:18.596446 7652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:36141 every 8 connection(s)
I20260924 00:50:18.598414 7585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
I20260924 00:50:18.603219 7653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:18.609892 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 7585
I20260924 00:50:18.649720 7653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:50:18 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
Sep 24 00:50:18 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211018, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.61@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:18.671722 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.618394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.0.222.61:36141)
0924 00:50:18.623256 (+ 4862us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:18.623452 (+ 196us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:18.624017 (+ 565us) server_negotiation.cc:407] Connection header received
0924 00:50:18.645537 (+ 21520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:18.645774 (+ 237us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:18.649414 (+ 3640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:18.652100 (+ 2686us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:18.653701 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.655646 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.657039 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.658332 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.658538 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.663194 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:18.663201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:18.663214 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:18.663260 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:18.666986 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.667928 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.667937 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.667948 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.668136 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.668604 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.668608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.668610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.669152 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:18.669507 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:18.669946 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 00:50:18.670361 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3081,"thread_start_us":1903,"threads_started":1}
I20260924 00:50:18.673599 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.655632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46206)
0924 00:50:18.657726 (+ 2094us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:18.657776 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:18.657948 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:18.658917 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:18.658925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:18.659903 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:18.660331 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.660344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.661667 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.661672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.662810 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.662823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.663706 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.663712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.663940 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.664460 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:18.664484 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:18.667365 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:18.670824 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.670834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.670842 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.671387 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.671745 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.671749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.671753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.671939 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.672578 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:18.672585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:18.672755 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 00:50:18.672985 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 00:50:18.673283 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1773,"thread_start_us":1585,"threads_started":1}
I20260924 00:50:18.673804 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.655649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.0.222.62:42943)
0924 00:50:18.658412 (+ 2763us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:18.658419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:18.658447 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:50:18.658527 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:18.658533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:18.658619 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:18.658753 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:18.660499 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.661528 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.662960 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.663521 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.663630 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.667575 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:18.667580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:18.667583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:18.667622 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:18.670691 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.671486 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.671489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.671492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.671637 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.672077 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.672082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.672085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.672453 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:18.672582 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:18.673120 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 00:50:18.673415 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2632,"thread_start_us":2505,"threads_started":1}
I20260924 00:50:18.675379 7653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:50:18 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211018, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.61@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
I20260924 00:50:18.688410 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.676141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.0.1:39380)
0924 00:50:18.676330 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:18.676348 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:18.676443 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:18.676760 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:18.676763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:18.677137 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:18.677339 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.677349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.678393 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.678397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.679334 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.679346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.680056 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.680060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.680233 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.680790 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:18.680807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:18.683045 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:18.686114 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.686118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.686120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.686525 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.686957 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.686961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.686963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.687093 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.687604 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:18.687607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:18.687751 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 00:50:18.687861 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 00:50:18.688106 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 00:50:18.688539 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.676384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.222.61:36141)
0924 00:50:18.676544 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:18.676548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:18.676562 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:50:18.676600 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:18.676603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:18.676654 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:18.676725 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:18.677467 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.678248 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.679461 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.679912 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.680014 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.683221 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:18.683226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:18.683230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:18.683284 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:18.685991 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.686648 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.686652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.686655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.686869 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.687192 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.687196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.687198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.687515 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:18.687624 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:18.687994 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 00:50:18.688237 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":67}
I20260924 00:50:18.694018 7653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Bootstrap starting.
I20260924 00:50:18.697568 7653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:18.698809 7653 log.cc:824] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:18.702060 7653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: No bootstrap required, opened a new log
I20260924 00:50:18.710080 7653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } }
I20260924 00:50:18.710458 7653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:18.710568 7653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e488b41d2c04decbea7acdaa202c91e, State: Initialized, Role: FOLLOWER
I20260924 00:50:18.711200 7653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } }
I20260924 00:50:18.712244 7660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } } }
I20260924 00:50:18.712613 7660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:50:18.713289 7653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 00:50:18.723958 7671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:50:18.724136 7671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:50:18.726111 7585 master_runner.cc:428] Detected that this master 6e488b41d2c04decbea7acdaa202c91e is joining an existing cluster
I20260924 00:50:18.726248 7585 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.222.61:36141
WARNING: no policy specified for oryx/127.0.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.60@KRBTEST.COM" created.
I20260924 00:50:18.759766 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.731326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.222.61:36141)
0924 00:50:18.731749 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:18.731755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:18.743636 (+ 11881us) server_negotiation.cc:407] Connection header received
0924 00:50:18.743770 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:18.743774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:18.743825 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:18.743903 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:18.745081 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.746311 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.748592 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.749425 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.749528 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.752493 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:18.752500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:18.752505 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:18.752538 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:18.756323 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.757415 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.757420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.757423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.757630 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.757974 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.757979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.757983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.758419 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:18.758550 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:18.758981 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 00:50:18.759353 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":270}
I20260924 00:50:18.760433 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.730617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.0.1:39396)
0924 00:50:18.743537 (+ 12920us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:18.743552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:18.743674 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:18.743935 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:18.743939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:18.744613 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:18.744949 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.744958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.746571 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.746576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.748401 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.748422 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.749870 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.749876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.750053 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.750992 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:18.751014 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:18.752289 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:18.756508 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.756519 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.756523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.757100 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.757730 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.757733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.757735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.757870 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.758525 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:18.758529 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:18.758671 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 00:50:18.758799 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 00:50:18.759954 (+ 1155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12791,"thread_start_us":12571,"threads_started":1}
I20260924 00:50:18.766748 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.731519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.0.222.62:42943)
0924 00:50:18.731731 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:18.731738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:18.742412 (+ 10674us) server_negotiation.cc:407] Connection header received
0924 00:50:18.742639 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:18.742645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:18.742807 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:18.742943 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:18.746644 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.747795 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.749975 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.750715 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.750837 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.753685 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:18.753692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:18.753696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:18.753753 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:18.757395 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.758279 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.758285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.758289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.758459 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.758951 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.758957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.758960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.759483 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:18.759624 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:18.766044 (+ 6420us) server_negotiation.cc:299] Negotiation successful
0924 00:50:18.766315 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 00:50:18.766803 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.730577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46218)
0924 00:50:18.742244 (+ 11667us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:18.742272 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:18.742498 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:18.745318 (+ 2820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:18.745323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:18.746066 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:18.746464 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.746475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.748020 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.748025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.749806 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.749819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.751179 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.751185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.751388 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.752026 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:18.752122 (+ 96us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:18.753442 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:18.757550 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.757556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.757561 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.758079 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.758603 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.758608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.758612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.758803 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.765426 (+ 6623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:18.765432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:18.765657 (+ 225us) client_negotiation.cc:769] Sending connection context
0924 00:50:18.765874 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 00:50:18.766255 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11443,"thread_start_us":12022,"threads_started":1}
Sep 24 00:50:18 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211016, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.60.keytab.
Entry for principal oryx/127.0.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.60.keytab.
I20260924 00:50:18.792711 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:35737
--webserver_interface=127.0.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.60.keytab
--principal=oryx/127.0.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42943,127.0.222.61:36141,127.0.222.60:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:50:18.793442 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.773370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.0.1:39398)
0924 00:50:18.776416 (+ 3046us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:18.776455 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:18.776728 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:18.777250 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:18.777258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:18.778129 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:18.778394 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.778405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.779445 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.779450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.780541 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.780554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.781494 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.781502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.781697 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.782321 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:18.782350 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:18.786317 (+ 3967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:18.789977 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.789993 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.790002 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.790674 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.791142 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.791147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.791151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.791326 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.792082 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:18.792091 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:18.792330 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 00:50:18.792566 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 00:50:18.792977 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2819,"thread_start_us":2607,"threads_started":1}
I20260924 00:50:18.794204 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.773375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.0.222.61:36141)
0924 00:50:18.773560 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:18.773567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:18.776552 (+ 2985us) server_negotiation.cc:407] Connection header received
0924 00:50:18.776943 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:18.776947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:18.777043 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:18.777136 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:18.778553 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.779316 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.780700 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.781314 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.781428 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.786591 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:18.786597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:18.786601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:18.786636 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:18.789823 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.790816 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.790820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.790823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.790986 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.791469 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.791475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.791479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.791920 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:18.792077 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:18.792706 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 00:50:18.793236 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 00:50:18.796272 7118 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.222.61:36141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.222.61"
port: 36141
}
attrs {
promote: true
}
}
dest_uuid: "1bff8d21582e43049659eaed39fe3937"
cas_config_opid_index: -1
I20260924 00:50:18.797544 7118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } }
I20260924 00:50:18.813614 7628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:18.818583 7628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Refusing update from remote peer 1bff8d21582e43049659eaed39fe3937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260924 00:50:18.818872 7607 tablet.cc:2369] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:50:18.821657 7685 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 00:50:18.820585 7686 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:18.834589 7686 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.222.61:36141
I20260924 00:50:18.836087 7585 master.cc:537] Master@127.0.222.61:36141 shutting down...
I20260924 00:50:18.837345 7685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:18.837881 7685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:18.874711 7628 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:18.898751 7694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:18.899190 7694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: LEARNER
I20260924 00:50:18.899293 7660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bff8d21582e43049659eaed39fe3937. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:18.899591 7660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: LEARNER
I20260924 00:50:18.899729 7690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:18.900133 7690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: LEARNER
I20260924 00:50:18.903047 7585 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 LEARNER]: Raft consensus shutting down.
I20260924 00:50:18.903600 7585 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 LEARNER]: Raft consensus is shut down!
I20260924 00:50:18.903786 7585 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: stopping tablet replica
I20260924 00:50:18.912138 7585 master.cc:552] Master@127.0.222.61:36141 shutdown complete.
I20260924 00:50:18.912251 7585 master_runner.cc:305] Clearing existing system tablet
I20260924 00:50:18.912767 7585 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:18.912910 7585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:18.916474 7585 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 00:50:18.918398 7696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:18.918828 7585 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:18.919329 7585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 00:50:18.919481 7585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
uuid: "6e488b41d2c04decbea7acdaa202c91e"
format_stamp: "Formatted at 2026-09-24 00:50:18 on dist-test-slave-rtvz"
server_key: "d8b3cccf9c8d0f4eb0077893cff944a1"
server_key_iv: "a8851c141480076cc9b1e14bbc31cbd1"
server_key_version: "encryptionkey@0"
I20260924 00:50:18.924606 7585 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:18.933364 7585 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 00:50:18.933763 7585 log.cc:1196] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 00:50:18.934402 7585 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Deleting consensus metadata
I20260924 00:50:18.934788 7585 master_runner.cc:315] Copying system tablet from 127.0.222.62:42943
I20260924 00:50:18.937827 7585 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.222.62:42943
I20260924 00:50:18.956168 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.939009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46228)
0924 00:50:18.939627 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:18.939649 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:18.939907 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:18.940527 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:18.940532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:18.941137 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:18.941510 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.941525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.943162 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.943168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.944767 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:18.944788 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.946030 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.946037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:18.946230 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.947101 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:18.947128 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:18.948427 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:18.952673 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.952678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.952682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.953190 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.953740 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:18.953745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:18.953748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:18.953963 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:18.954802 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:18.954807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:18.955057 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 00:50:18.955230 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 00:50:18.955613 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":375,"threads_started":1}
I20260924 00:50:18.957425 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:18.939306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.0.222.62:42943)
0924 00:50:18.939491 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:18.939498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:18.939774 (+ 276us) server_negotiation.cc:407] Connection header received
0924 00:50:18.940102 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:18.940109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:18.940199 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:18.940274 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:18.941713 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.942987 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.944962 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:18.945795 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:18.945991 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:18.948646 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:18.948654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:18.948659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:18.948701 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:18.952516 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.953342 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.953347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.953358 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.953601 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:18.954080 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:18.954085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:18.954088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:18.954538 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:18.954735 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:18.956704 (+ 1969us) server_negotiation.cc:299] Negotiation successful
0924 00:50:18.956984 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:50:18.958328 7139 tablet_copy_service.cc:140] P 1bff8d21582e43049659eaed39fe3937: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6e488b41d2c04decbea7acdaa202c91e ({username='oryx', principal='oryx/127.0.222.61@KRBTEST.COM'} at 127.0.0.1:46228)
I20260924 00:50:18.958623 7139 tablet_copy_service.cc:161] P 1bff8d21582e43049659eaed39fe3937: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6e488b41d2c04decbea7acdaa202c91e at {username='oryx', principal='oryx/127.0.222.61@KRBTEST.COM'} at 127.0.0.1:46228: session id = 6e488b41d2c04decbea7acdaa202c91e-00000000000000000000000000000000
I20260924 00:50:18.963457 7139 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:50:18.966463 7585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 00:50:18.974254 7585 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: tablet copy: Starting download of 0 data blocks...
I20260924 00:50:18.974632 7585 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: tablet copy: Starting download of 1 WAL segments...
I20260924 00:50:18.978335 7585 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:50:18.983500 7139 tablet_copy_service.cc:342] P 1bff8d21582e43049659eaed39fe3937: Request end of tablet copy session 6e488b41d2c04decbea7acdaa202c91e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.222.61@KRBTEST.COM'} at 127.0.0.1:46228
I20260924 00:50:18.983794 7139 tablet_copy_service.cc:433] P 1bff8d21582e43049659eaed39fe3937: ending tablet copy session 6e488b41d2c04decbea7acdaa202c91e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6e488b41d2c04decbea7acdaa202c91e
I20260924 00:50:18.997490 7585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:19.000545 7703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:19.001062 7704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:19.003469 7706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:19.006521 7585 server_base.cc:1034] running on GCE node
I20260924 00:50:19.006915 7585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:19.007616 7585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:19.008886 7585 hybrid_clock.cc:648] HybridClock initialized: now 1790211019008796 us; error 49 us; skew 500 ppm
I20260924 00:50:19.009159 7585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:19 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:19.011250 7585 init.cc:377] Logged in from keytab as oryx/127.0.222.61@KRBTEST.COM (short username oryx)
I20260924 00:50:19.022428 7585 webserver.cc:460] Webserver started at http://127.0.222.61:43467/ using document root <none> and password file <none>
I20260924 00:50:19.022966 7585 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:19.023156 7585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:19.026681 7585 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 00:50:19.036338 7713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:19.036798 7585 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.005s
I20260924 00:50:19.036972 7585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
uuid: "6e488b41d2c04decbea7acdaa202c91e"
format_stamp: "Formatted at 2026-09-24 00:50:18 on dist-test-slave-rtvz"
server_key: "d8b3cccf9c8d0f4eb0077893cff944a1"
server_key_iv: "a8851c141480076cc9b1e14bbc31cbd1"
server_key_version: "encryptionkey@0"
I20260924 00:50:19.037141 7585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:19.046270 7580 heartbeater.cc:503] Master 127.0.222.62:42943 was elected leader, sending a full tablet report...
I20260924 00:50:19.065851 7585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:19.070070 7585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:19.118427 7585 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:36141
I20260924 00:50:19.118432 7765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:36141 every 8 connection(s)
I20260924 00:50:19.119760 7585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
I20260924 00:50:19.127177 7766 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:50:19.131018 7766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Bootstrap starting.
I20260924 00:50:19.147099 7766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:50:19.147863 7766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: Bootstrap complete.
I20260924 00:50:19.149243 7766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } }
I20260924 00:50:19.149554 7766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6e488b41d2c04decbea7acdaa202c91e, State: Initialized, Role: LEARNER
I20260924 00:50:19.149930 7766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } }
I20260924 00:50:19.154498 7766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:19.154042 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:19.154814 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: LEARNER
I20260924 00:50:19.177628 7780 catalog_manager.cc:1259] Loaded cluster ID: 7981b32f7ae2469a905d7dc221204823
I20260924 00:50:19.177736 7780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: loading cluster ID for follower catalog manager: success
I20260924 00:50:19.181106 7780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: acquiring CA information for follower catalog manager: success
I20260924 00:50:19.183983 7780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:50:19.240520 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:19.241452 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:19.260133 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.241387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.0.1:39406)
0924 00:50:19.242052 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:19.242075 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:19.242246 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:19.244397 (+ 2151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:19.244403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:19.244451 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:19.245038 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.245053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.247755 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.247771 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.251220 (+ 3449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.251269 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.256609 (+ 5340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.256628 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.257771 (+ 1143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.257846 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 00:50:19.258045 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 00:50:19.258565 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504}
I20260924 00:50:19.261667 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.242395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.222.61:36141)
0924 00:50:19.243646 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.243665 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.243748 (+ 83us) server_negotiation.cc:407] Connection header received
0924 00:50:19.243896 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.243908 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.244044 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.244277 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:19.245267 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.247473 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.251699 (+ 4226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.255956 (+ 4257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.256509 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.257633 (+ 1124us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.258084 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.258677 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"thread_start_us":792,"threads_started":1}
I20260924 00:50:19.265820 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bff8d21582e43049659eaed39fe3937. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:19.267151 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: LEARNER
I20260924 00:50:19.269811 7685 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: attempting to promote NON_VOTER 6e488b41d2c04decbea7acdaa202c91e to VOTER
I20260924 00:50:19.271991 7685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } }
I20260924 00:50:19.276885 7741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 LEARNER]: Refusing update from remote peer 1bff8d21582e43049659eaed39fe3937: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 00:50:19.278156 7685 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:50:19.289717 7686 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } }
I20260924 00:50:19.292898 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } }
I20260924 00:50:19.293191 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:19.293246 7684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } }
I20260924 00:50:19.293569 7684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:19.294709 7741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } }
I20260924 00:50:19.298558 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } }
I20260924 00:50:19.298765 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:50:19.427620 7682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:19.428130 7682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:19.428392 7682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:19.443383 7682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:19.443603 7682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:19.443720 7682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:19.443868 7682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:19.443965 7682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:19.459242 7682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42943,127.0.222.61:36141,127.0.222.60:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.60.keytab
--principal=oryx/127.0.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:19.460935 7682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:19.462546 7682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:19.478495 7787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:19.478789 7790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:19.478974 7788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:19.480511 7682 server_base.cc:1034] running on GCE node
I20260924 00:50:19.481557 7682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:19.483202 7682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:19.484453 7682 hybrid_clock.cc:648] HybridClock initialized: now 1790211019484341 us; error 70 us; skew 500 ppm
I20260924 00:50:19.484895 7682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:19 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:19.489420 7682 init.cc:377] Logged in from keytab as oryx/127.0.222.60@KRBTEST.COM (short username oryx)
I20260924 00:50:19.491513 7682 webserver.cc:460] Webserver started at http://127.0.222.60:43439/ using document root <none> and password file <none>
I20260924 00:50:19.492369 7682 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:19.492555 7682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:19.492923 7682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:19.496995 7682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data/instance:
uuid: "3986ed3b261f467fa0d79609b31b073c"
format_stamp: "Formatted at 2026-09-24 00:50:19 on dist-test-slave-rtvz"
server_key: "effc3ede438bfda44b45904146c543e6"
server_key_iv: "17efba5085771be8b7cda43cb51cb908"
server_key_version: "encryptionkey@0"
I20260924 00:50:19.498153 7682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal/instance:
uuid: "3986ed3b261f467fa0d79609b31b073c"
format_stamp: "Formatted at 2026-09-24 00:50:19 on dist-test-slave-rtvz"
server_key: "effc3ede438bfda44b45904146c543e6"
server_key_iv: "17efba5085771be8b7cda43cb51cb908"
server_key_version: "encryptionkey@0"
I20260924 00:50:19.505225 7682 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260924 00:50:19.510658 7797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:19.512416 7682 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 00:50:19.512681 7682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
uuid: "3986ed3b261f467fa0d79609b31b073c"
format_stamp: "Formatted at 2026-09-24 00:50:19 on dist-test-slave-rtvz"
server_key: "effc3ede438bfda44b45904146c543e6"
server_key_iv: "17efba5085771be8b7cda43cb51cb908"
server_key_version: "encryptionkey@0"
I20260924 00:50:19.512988 7682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:19.540477 7682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:19.546052 7682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:19.546450 7682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:19.584872 7682 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:35737
I20260924 00:50:19.584949 7848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:35737 every 8 connection(s)
I20260924 00:50:19.586539 7682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
I20260924 00:50:19.591598 7849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:19.592047 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 7682
I20260924 00:50:19.604286 7849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:50:19 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
Sep 24 00:50:19 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:19.624087 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.594742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.0.222.60:35737)
0924 00:50:19.596151 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.596164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.596224 (+ 60us) server_negotiation.cc:407] Connection header received
0924 00:50:19.598629 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.598656 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.599188 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.599597 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.601886 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.603625 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.605388 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.606699 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.606942 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.613740 (+ 6798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.613750 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.613764 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.613805 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.618189 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.619358 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.619365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.619372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.619624 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.620143 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.620149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.620152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.620976 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.621293 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.621857 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.622352 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"spinlock_wait_cycles":17664,"thread_start_us":719,"threads_started":1}
I20260924 00:50:19.630050 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.608019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.0.222.62:42943)
0924 00:50:19.608768 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.608773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.609787 (+ 1014us) server_negotiation.cc:407] Connection header received
0924 00:50:19.610039 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.610043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.610110 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.610203 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.612031 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.613700 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.615929 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.616692 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.616817 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.621970 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.621976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.621981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.622023 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.626439 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.627608 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.627617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.627621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.627883 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.628263 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.628266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.628268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.628616 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.628731 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.629427 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.629633 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":478,"threads_started":1}
I20260924 00:50:19.630079 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.608099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46238)
0924 00:50:19.609642 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:19.609689 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:19.609902 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:19.610288 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:19.610297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:19.611342 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:19.611867 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.611879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.613888 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.613893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.615725 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.615776 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.616995 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.617006 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.617187 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.618027 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:19.618052 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:19.621735 (+ 3683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:19.626595 (+ 4860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.626604 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.626611 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.627444 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.627991 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.627995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.627997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.628175 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.628756 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:19.628762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:19.629037 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 00:50:19.629319 (+ 282us) client_negotiation.cc:240] Negotiation successful
0924 00:50:19.629599 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"thread_start_us":1008,"threads_started":1}
I20260924 00:50:19.631598 7849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:50:19 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
I20260924 00:50:19.649004 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.632576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.0.1:39416)
0924 00:50:19.632778 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:19.632796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:19.632902 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:19.633157 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:19.633160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:19.633662 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:19.633910 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.633921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.635233 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.635242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.636584 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.636597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.637788 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.637795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.638015 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.638768 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:19.638787 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:19.641808 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:19.645946 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.645950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.645953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.646583 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.647153 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.647157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.647160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.647401 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.648083 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:19.648086 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:19.648250 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 00:50:19.648382 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 00:50:19.648693 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 00:50:19.649081 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.632440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.0.222.61:36141)
0924 00:50:19.632599 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.632604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.632904 (+ 300us) server_negotiation.cc:407] Connection header received
0924 00:50:19.632980 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.632985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.633054 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.633137 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.634053 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.635039 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.636737 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.637504 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.637627 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.642046 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.642051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.642055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.642102 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.645769 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.646842 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.646847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.646849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.647055 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.647490 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.647494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.647496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.647978 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.648140 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.648491 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.648718 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:50:19.650272 7849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:50:19 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211019, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
I20260924 00:50:19.666287 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.650943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:35737 (local address 127.0.0.1:56810)
0924 00:50:19.651192 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:19.651207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:19.651296 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:19.651536 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:19.651539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:19.652013 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:19.652276 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.652286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.653516 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.653519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.654673 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.654686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.655680 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.655685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.655995 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.656617 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:19.656635 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:19.659253 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:19.663250 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.663255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.663258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.663820 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.664238 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.664242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.664245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.664507 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.665077 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:19.665080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:19.665357 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 00:50:19.665484 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 00:50:19.665771 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 00:50:19.666296 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.650943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.0.222.60:35737)
0924 00:50:19.651122 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.651126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.651267 (+ 141us) server_negotiation.cc:407] Connection header received
0924 00:50:19.651382 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.651385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.651442 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.651515 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.652395 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.653403 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.654837 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.655500 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.655644 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.659511 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.659516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.659520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.659552 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.663068 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.663932 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.663936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.663938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.664118 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.664600 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.664603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.664605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.664952 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.665062 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.665591 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.665956 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":47}
I20260924 00:50:19.673172 7849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Bootstrap starting.
I20260924 00:50:19.678213 7849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:19.680121 7849 log.cc:824] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:19.683923 7849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: No bootstrap required, opened a new log
I20260924 00:50:19.695013 7849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } }
I20260924 00:50:19.695636 7849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:19.695801 7849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3986ed3b261f467fa0d79609b31b073c, State: Initialized, Role: FOLLOWER
I20260924 00:50:19.696471 7849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } }
I20260924 00:50:19.697511 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } } }
I20260924 00:50:19.698046 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:50:19.698750 7849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: configured and running, proceeding with master startup.
W20260924 00:50:19.712891 7867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:50:19.713104 7867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:50:19.717087 7682 master_runner.cc:428] Detected that this master 3986ed3b261f467fa0d79609b31b073c is joining an existing cluster
I20260924 00:50:19.717320 7682 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.222.60:35737
I20260924 00:50:19.732817 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 7182
Sep 24 00:50:19 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.1@KRBTEST.COM
I20260924 00:50:19.755028 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.723900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.0.222.62:42943)
0924 00:50:19.724092 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.724098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.731139 (+ 7041us) server_negotiation.cc:407] Connection header received
0924 00:50:19.731621 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.731628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.731762 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.731848 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.736039 (+ 4191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.737469 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.740367 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.741252 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.741363 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.746635 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.746643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.746648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.746694 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.751314 (+ 4620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.752310 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.752317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.752321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.752496 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.753013 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.753018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.753022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.753473 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.753652 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.754362 (+ 710us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.754642 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:50:19.756201 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.724409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46248)
0924 00:50:19.731001 (+ 6592us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:19.731024 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:19.731439 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:19.732064 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:19.732069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:19.733805 (+ 1736us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:19.735812 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.735829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.737817 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.737823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.740148 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.740167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.741693 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.741700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.741937 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.743093 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:19.743121 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:19.744454 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:19.751491 (+ 7037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.751497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.751501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.752108 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.752659 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.752664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.752668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.752858 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.753743 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:19.753748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:19.754033 (+ 285us) client_negotiation.cc:769] Sending connection context
0924 00:50:19.755223 (+ 1190us) client_negotiation.cc:240] Negotiation successful
0924 00:50:19.755692 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6389,"thread_start_us":17908,"threads_started":1}
I20260924 00:50:19.761830 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.726209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.0.222.60:35737)
0924 00:50:19.726388 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.726394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.742797 (+ 16403us) server_negotiation.cc:407] Connection header received
0924 00:50:19.742936 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.742941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.743079 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.743200 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.744755 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.745855 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.747803 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.748326 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.748417 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.752479 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.752485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.752490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.752534 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.757740 (+ 5206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.758935 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.758940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.758942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.759100 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.759541 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.759546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.759550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.760015 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.760157 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.760627 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.760923 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 00:50:19.762733 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.723846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.0.222.61:36141)
0924 00:50:19.724051 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.724057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.725973 (+ 1916us) server_negotiation.cc:407] Connection header received
0924 00:50:19.726348 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.726356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.726450 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.726528 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.729963 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.731349 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.749415 (+ 18066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.750078 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.750194 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.754853 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.754860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.754865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.754914 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.758141 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.760290 (+ 2149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.760297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.760300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.760464 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.760979 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.760984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.760986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.761385 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.761519 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.762122 (+ 603us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.762375 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 00:50:19.762776 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.726502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:35737 (local address 127.0.0.1:56814)
0924 00:50:19.742668 (+ 16166us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:19.742692 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:19.742834 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:19.743387 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:19.743392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:19.744246 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:19.744604 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.744617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.746105 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.746110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.747663 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.747679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.748738 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.748745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.748895 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.750469 (+ 1574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:19.750490 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:19.751608 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:19.758296 (+ 6688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.758302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.758306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.758813 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.759239 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.759244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.759248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.759399 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.760208 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:19.760216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:19.760387 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 00:50:19.762049 (+ 1662us) client_negotiation.cc:240] Negotiation successful
0924 00:50:19.762348 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15989,"thread_start_us":4675,"threads_started":1}
I20260924 00:50:19.763710 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.724359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.0.1:39422)
0924 00:50:19.725832 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:19.725856 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:19.726147 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:19.726769 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:19.726774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:19.728709 (+ 1935us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:19.729722 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.729745 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.744633 (+ 14888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.744639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.746368 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.746383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.751855 (+ 5472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.751862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.752089 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.753347 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:19.753371 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:19.754562 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:19.759571 (+ 5009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.759575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.759579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.760137 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.760606 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.760610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.760613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.760855 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.761597 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:19.761601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:19.761810 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 00:50:19.762984 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0924 00:50:19.763297 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"thread_start_us":1025,"threads_started":1}
I20260924 00:50:19.766963 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.734222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.0.222.1:46755)
0924 00:50:19.735405 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.735417 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.735451 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:50:19.735538 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.735549 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.735917 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.738431 (+ 2514us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.738846 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.741479 (+ 2633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.743499 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.744434 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.750733 (+ 6299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.751057 (+ 324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.751074 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.751096 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.751144 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.763456 (+ 12312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.764614 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.764623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.764626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.764806 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.765291 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.765296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.765300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.765654 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.765977 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.766332 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.766608 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":556,"threads_started":1}
I20260924 00:50:19.768187 7306 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 24 00:50:19 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211016, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
I20260924 00:50:19.776786 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.762722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.0.222.60:35737)
0924 00:50:19.762910 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:19.762915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:19.762929 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:50:19.763014 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:19.763019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:19.763122 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:19.764119 (+ 997us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:19.764297 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.765037 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.766399 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.766935 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.767037 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.771117 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:19.771123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:19.771128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:19.771168 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:19.774321 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.774944 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.774950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.774954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.775069 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:19.775416 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:19.775421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:19.775425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:19.775703 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:19.775848 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:19.776224 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 00:50:19.776432 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 00:50:19.776799 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:19.761950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:35737 (local address 127.0.0.1:56824)
0924 00:50:19.762411 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:19.762432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:19.762536 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:19.763287 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:19.763292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:19.763759 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:19.763976 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.763989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.765216 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.765222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.766228 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:19.766241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:19.767075 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:19.767081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:19.767235 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:19.767864 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:19.767895 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:19.770917 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:19.774463 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.774468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.774473 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.774831 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.775174 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:19.775179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:19.775183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:19.775307 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:19.775818 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:19.775822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:19.775982 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 00:50:19.776119 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 00:50:19.776416 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337}
I20260924 00:50:19.778432 7118 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.222.60:35737: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3986ed3b261f467fa0d79609b31b073c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.222.60"
port: 35737
}
attrs {
promote: true
}
}
dest_uuid: "1bff8d21582e43049659eaed39fe3937"
cas_config_opid_index: 6
I20260924 00:50:19.779115 7118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } }
I20260924 00:50:19.781394 7741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Refusing update from remote peer 1bff8d21582e43049659eaed39fe3937: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 00:50:19.782021 7684 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 00:50:19.782969 7824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:19.784371 7685 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3986ed3b261f467fa0d79609b31b073c (127.0.222.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:19.787128 7824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 FOLLOWER]: Refusing update from remote peer 1bff8d21582e43049659eaed39fe3937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 00:50:19.788017 7686 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:19.789525 7741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3986ed3b261f467fa0d79609b31b073c (127.0.222.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:19.793381 7685 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.222.60:35737
I20260924 00:50:19.794185 7682 master.cc:537] Master@127.0.222.60:35737 shutting down...
I20260924 00:50:19.795781 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:19.796090 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:19.795975 7684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:19.796277 7684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:19.797787 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:19.798251 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:50:19.799773 7803 tablet.cc:2369] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:50:19.807190 7823 raft_consensus.cc:858] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } }
W20260924 00:50:19.807386 7823 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } }
I20260924 00:50:19.837832 7823 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:19.861245 7884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:19.862035 7884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 00:50:19.862226 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bff8d21582e43049659eaed39fe3937. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:19.862534 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 00:50:19.863844 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: NON_VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: true } } }
I20260924 00:50:19.864473 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 00:50:19.866235 7685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:19.866803 7685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
W20260924 00:50:19.869807 7106 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 00:50:19.873792 7106 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 -> Peer 3986ed3b261f467fa0d79609b31b073c (127.0.222.60:35737): Couldn't send request to peer 3986ed3b261f467fa0d79609b31b073c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:50:19.874862 7682 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260924 00:50:19.875743 7682 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260924 00:50:19.876225 7682 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: stopping tablet replica
I20260924 00:50:19.899674 7682 master.cc:552] Master@127.0.222.60:35737 shutdown complete.
I20260924 00:50:19.900056 7682 master_runner.cc:305] Clearing existing system tablet
I20260924 00:50:19.901769 7682 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:19.902801 7682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:19.915649 7682 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.008s sys 0.000s
I20260924 00:50:19.922454 7887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:19.924958 7682 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:19.927136 7682 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.004s
I20260924 00:50:19.927978 7682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
uuid: "3986ed3b261f467fa0d79609b31b073c"
format_stamp: "Formatted at 2026-09-24 00:50:19 on dist-test-slave-rtvz"
server_key: "effc3ede438bfda44b45904146c543e6"
server_key_iv: "17efba5085771be8b7cda43cb51cb908"
server_key_version: "encryptionkey@0"
I20260924 00:50:19.950030 7682 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:19.984047 7682 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 00:50:19.984830 7682 log.cc:1196] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 00:50:19.987112 7682 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Deleting consensus metadata
I20260924 00:50:19.987736 7682 master_runner.cc:315] Copying system tablet from 127.0.222.62:42943
I20260924 00:50:19.999557 7682 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.222.62:42943
I20260924 00:50:20.000813 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 7182
I20260924 00:50:20.016412 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.000602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.0.222.62:42943)
0924 00:50:20.000886 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:20.000893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:20.001596 (+ 703us) server_negotiation.cc:407] Connection header received
0924 00:50:20.002009 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:20.002016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:20.002109 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:20.002229 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:20.003588 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.004798 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.006654 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.007149 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.007271 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.009463 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:20.009468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:20.009471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:20.009507 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:20.013137 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.013986 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.013992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.013996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.014262 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.014694 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.014700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.014704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.015143 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:20.015299 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:20.015774 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 00:50:20.016027 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260924 00:50:20.016434 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.000594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46252)
0924 00:50:20.001480 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:20.001505 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:20.001753 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:20.002268 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:20.002273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:20.003041 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:20.003403 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.003427 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.004975 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.004981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.006508 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.006526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.007329 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.007335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.007537 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.008210 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:20.008229 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:20.009161 (+ 932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:20.013345 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:20.013351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:20.013356 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:20.013831 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:20.014376 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:20.014381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:20.014385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:20.014550 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:20.015273 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:20.015278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:20.015455 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 00:50:20.015576 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 00:50:20.015987 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":426,"threads_started":1}
I20260924 00:50:20.017460 7139 tablet_copy_service.cc:140] P 1bff8d21582e43049659eaed39fe3937: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3986ed3b261f467fa0d79609b31b073c ({username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:46252)
I20260924 00:50:20.017666 7139 tablet_copy_service.cc:161] P 1bff8d21582e43049659eaed39fe3937: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3986ed3b261f467fa0d79609b31b073c at {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:46252: session id = 3986ed3b261f467fa0d79609b31b073c-00000000000000000000000000000000
I20260924 00:50:20.020632 7139 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:50:20.025844 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 00:50:20.031095 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:46755
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:42943
--webserver_port=46707
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141,127.0.222.60:35737 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:50:20.033962 7682 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: tablet copy: Starting download of 0 data blocks...
I20260924 00:50:20.034344 7682 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: tablet copy: Starting download of 1 WAL segments...
I20260924 00:50:20.037041 7682 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:50:20.041350 7139 tablet_copy_service.cc:342] P 1bff8d21582e43049659eaed39fe3937: Request end of tablet copy session 3986ed3b261f467fa0d79609b31b073c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:46252
I20260924 00:50:20.041539 7139 tablet_copy_service.cc:433] P 1bff8d21582e43049659eaed39fe3937: ending tablet copy session 3986ed3b261f467fa0d79609b31b073c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3986ed3b261f467fa0d79609b31b073c
I20260924 00:50:20.058004 7682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:20.061762 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:20.063071 7896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:20.074223 7682 server_base.cc:1034] running on GCE node
W20260924 00:50:20.077513 7898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:20.077998 7682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:20.078647 7682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:20.079842 7682 hybrid_clock.cc:648] HybridClock initialized: now 1790211020079795 us; error 37 us; skew 500 ppm
I20260924 00:50:20.080097 7682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:20 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:20.081996 7682 init.cc:377] Logged in from keytab as oryx/127.0.222.60@KRBTEST.COM (short username oryx)
I20260924 00:50:20.086488 7682 webserver.cc:460] Webserver started at http://127.0.222.60:45933/ using document root <none> and password file <none>
I20260924 00:50:20.086812 7682 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:20.086935 7682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:20.089452 7682 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 00:50:20.091166 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:20.091660 7682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:50:20.091769 7682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
uuid: "3986ed3b261f467fa0d79609b31b073c"
format_stamp: "Formatted at 2026-09-24 00:50:19 on dist-test-slave-rtvz"
server_key: "effc3ede438bfda44b45904146c543e6"
server_key_iv: "17efba5085771be8b7cda43cb51cb908"
server_key_version: "encryptionkey@0"
I20260924 00:50:20.091958 7682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:20.113153 7682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:20.117604 7682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:20.164487 7682 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:35737
I20260924 00:50:20.164683 7957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:35737 every 8 connection(s)
I20260924 00:50:20.165558 7682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
I20260924 00:50:20.178496 7958 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:50:20.182866 7958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Bootstrap starting.
I20260924 00:50:20.196154 7958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:50:20.196658 7958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Bootstrap complete.
I20260924 00:50:20.197650 7958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } }
I20260924 00:50:20.197937 7958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3986ed3b261f467fa0d79609b31b073c, State: Initialized, Role: LEARNER
I20260924 00:50:20.198163 7958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } }
I20260924 00:50:20.199020 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:20.199528 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: LEARNER
I20260924 00:50:20.199371 7958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:20.209975 7972 catalog_manager.cc:1259] Loaded cluster ID: 7981b32f7ae2469a905d7dc221204823
I20260924 00:50:20.210083 7972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: loading cluster ID for follower catalog manager: success
I20260924 00:50:20.212558 7972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: acquiring CA information for follower catalog manager: success
I20260924 00:50:20.214555 7972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:50:20.282847 7685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:20.283254 7685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:20.295609 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.283287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:35737 (local address 127.0.0.1:56828)
0924 00:50:20.283841 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:20.283865 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:20.284036 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:20.290167 (+ 6131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:20.290173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:20.290212 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:20.290446 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.290456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.291878 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.291885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.293410 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.293429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.294733 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.294739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.294928 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.294964 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 00:50:20.295060 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:50:20.295211 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":982,"threads_started":1}
I20260924 00:50:20.296025 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.283921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.0.222.60:35737)
0924 00:50:20.289747 (+ 5826us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:20.289752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:20.289778 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:50:20.289856 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:20.289859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:20.289947 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:20.290096 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:20.290621 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.291702 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.293575 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.294535 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.294671 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.294948 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:20.295235 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 00:50:20.295403 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5682,"thread_start_us":5485,"threads_started":1}
I20260924 00:50:20.296989 7933 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260924 00:50:20.297466 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bff8d21582e43049659eaed39fe3937. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: NON_VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: true } } }
I20260924 00:50:20.297745 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: LEARNER
I20260924 00:50:20.298221 7685 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: attempting to promote NON_VOTER 3986ed3b261f467fa0d79609b31b073c to VOTER
I20260924 00:50:20.298940 7685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } }
I20260924 00:50:20.301018 7933 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 LEARNER]: Refusing update from remote peer 1bff8d21582e43049659eaed39fe3937: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 00:50:20.301128 7741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Refusing update from remote peer 1bff8d21582e43049659eaed39fe3937: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 00:50:20.301796 7684 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 00:50:20.303812 7684 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260924 00:50:20.306317 7685 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3986ed3b261f467fa0d79609b31b073c (127.0.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:20.307650 7933 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3986ed3b261f467fa0d79609b31b073c (127.0.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:20.310401 7741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3986ed3b261f467fa0d79609b31b073c (127.0.222.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:20.314047 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:20.314314 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:50:20.314715 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:20.315037 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:20.315291 7686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:20.315515 7686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:20.314839 7685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:20.315939 7685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:20.320878 7976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bff8d21582e43049659eaed39fe3937" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:20.321367 7976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:50:20.578425 7894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:20.579149 7894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:20.579526 7894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:20.599942 7894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:20.600219 7894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:20.600833 7894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:20.633011 7894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:46755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=46707
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141,127.0.222.60:35737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:20.634610 7894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:20.636193 7894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:20.652329 7981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:20.652482 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:20.652643 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:20.653345 7894 server_base.cc:1034] running on GCE node
I20260924 00:50:20.654132 7894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:20.655367 7894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:20.656623 7894 hybrid_clock.cc:648] HybridClock initialized: now 1790211020656561 us; error 73 us; skew 500 ppm
I20260924 00:50:20.657096 7894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:20 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:20.660636 7894 init.cc:377] Logged in from keytab as oryx/127.0.222.1@KRBTEST.COM (short username oryx)
I20260924 00:50:20.662539 7894 webserver.cc:460] Webserver started at http://127.0.222.1:46707/ using document root <none> and password file <none>
I20260924 00:50:20.663305 7894 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:20.663447 7894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:20.670394 7894 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 00:50:20.674161 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:20.675910 7894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 00:50:20.676131 7894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "3359189faab547c5a64c98e926d18930"
format_stamp: "Formatted at 2026-09-24 00:50:16 on dist-test-slave-rtvz"
server_key: "ea750cf156f9961486d86b30d4a76ffc"
server_key_iv: "ede9c606637d09d79ce6a333a4c2b5d3"
server_key_version: "encryptionkey@0"
I20260924 00:50:20.677651 7894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:20.691751 7894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:20.696944 7894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:20.697288 7894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:20.698462 7894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:20.700250 7894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:20.700380 7894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:20.700501 7894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:20.700568 7894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:20.759959 7894 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:46755
I20260924 00:50:20.760115 8107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:46755 every 8 connection(s)
I20260924 00:50:20.761909 7894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:20.765970 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 7894
I20260924 00:50:20.766376 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 7318
Sep 24 00:50:20 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
Sep 24 00:50:20 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
Sep 24 00:50:20 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211020, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
Sep 24 00:50:20 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.2@KRBTEST.COM
I20260924 00:50:20.823388 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.766897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:33197 (local address 127.0.222.61:36141)
0924 00:50:20.796902 (+ 30005us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:20.796908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:20.796925 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:50:20.796975 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:20.796979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:20.797038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:20.797136 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:20.800650 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.802183 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.805746 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.806625 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.806712 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.813478 (+ 6766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:20.813488 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:20.813494 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:20.813543 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:20.818502 (+ 4959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.819777 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.819785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.819789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.819987 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.820724 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.820729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.820732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.821148 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:20.821329 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:20.822730 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0924 00:50:20.822974 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29797,"thread_start_us":29676,"threads_started":1}
I20260924 00:50:20.824973 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.767988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.0.222.2:40137)
0924 00:50:20.794052 (+ 26064us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:20.794382 (+ 330us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:20.794423 (+ 41us) server_negotiation.cc:407] Connection header received
0924 00:50:20.796179 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:20.796191 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:20.796529 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:20.796766 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:20.799591 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.801755 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.804483 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.805685 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.805795 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.815257 (+ 9462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:20.815280 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:20.815301 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:20.815347 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:20.820692 (+ 5345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.822015 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.822026 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.822029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.822217 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.822689 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.822694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.822698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.823156 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:20.823452 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:20.824094 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 00:50:20.824401 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25811,"thread_start_us":26100,"threads_started":1}
I20260924 00:50:20.826093 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.767205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:47527 (local address 127.0.222.62:42943)
0924 00:50:20.790011 (+ 22806us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:20.790020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:20.794448 (+ 4428us) server_negotiation.cc:407] Connection header received
0924 00:50:20.797189 (+ 2741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:20.797231 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:20.797322 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:20.797440 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:20.800610 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.802082 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.805769 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.806606 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.806708 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.814371 (+ 7663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:20.814381 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:20.814386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:20.814442 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:20.819360 (+ 4918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.820887 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.820922 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.820926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.821309 (+ 383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.822243 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.822249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.822253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.823053 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:20.823299 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:20.824819 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0924 00:50:20.825450 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22649,"thread_start_us":22828,"threads_started":1}
I20260924 00:50:20.826356 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.766939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.222.1:33197)
0924 00:50:20.788070 (+ 21131us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:20.788150 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:20.796602 (+ 8452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:20.797762 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:20.797776 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:20.798946 (+ 1170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:20.800356 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.800386 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.802368 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.802374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.805570 (+ 3196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.805585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.807188 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.807197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.807618 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.808715 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:20.808749 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:20.812964 (+ 4215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:20.818774 (+ 5810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:20.818786 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:20.818809 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:20.819588 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:20.820162 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:20.820167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:20.820171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:20.820570 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:20.821427 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:20.821435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:20.822250 (+ 815us) client_negotiation.cc:769] Sending connection context
0924 00:50:20.823721 (+ 1471us) client_negotiation.cc:240] Negotiation successful
0924 00:50:20.824335 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20591,"mutex_wait_us":327,"spinlock_wait_cycles":32000,"thread_start_us":26656,"threads_started":1}
I20260924 00:50:20.827409 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.766936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:47527)
0924 00:50:20.794177 (+ 27241us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:20.794222 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:20.797021 (+ 2799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:20.797762 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:20.797776 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:20.798951 (+ 1175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:20.800363 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.800386 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.802320 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.802325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.805570 (+ 3245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.805585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.807198 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.807205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.807561 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.808715 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:20.808758 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:20.814041 (+ 5283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:20.819541 (+ 5500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:20.819546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:20.819550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:20.820665 (+ 1115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:20.821637 (+ 972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:20.821643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:20.821648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:20.822016 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:20.823954 (+ 1938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:20.823962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:20.824307 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 00:50:20.826400 (+ 2093us) client_negotiation.cc:240] Negotiation successful
0924 00:50:20.826730 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27034,"mutex_wait_us":715,"spinlock_wait_cycles":57472,"thread_start_us":21830,"threads_started":1}
I20260924 00:50:20.828642 8110 heartbeater.cc:347] Connected to a master server at 127.0.222.61:36141
I20260924 00:50:20.829258 8110 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:20.829635 7441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:20.830801 8110 heartbeater.cc:511] Master 127.0.222.61:36141 requested a full tablet report, sending...
I20260924 00:50:20.828837 8108 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42943
I20260924 00:50:20.831900 8108 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:20.832615 8108 heartbeater.cc:511] Master 127.0.222.62:42943 requested a full tablet report, sending...
I20260924 00:50:20.832825 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.794806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:35737 (local address 127.0.222.1:35763)
0924 00:50:20.799603 (+ 4797us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:20.799644 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:20.799869 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:20.800436 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:20.800440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:20.801326 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:20.802036 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.802052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.803763 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.803768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.805879 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:20.805895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.807543 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.807556 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:20.807918 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.809483 (+ 1565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:20.809515 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:20.813329 (+ 3814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:20.819526 (+ 6197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:20.819532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:20.819536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:20.820169 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:20.829946 (+ 9777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:20.829952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:20.829956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:20.830531 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:20.831604 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:20.831609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:20.831758 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 00:50:20.831953 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 00:50:20.832244 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":2,"thread_start_us":728,"threads_started":1}
I20260924 00:50:20.834082 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:20.794550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:35763 (local address 127.0.222.60:35737)
0924 00:50:20.794778 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:20.794785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:20.799733 (+ 4948us) server_negotiation.cc:407] Connection header received
0924 00:50:20.800037 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:20.800042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:20.800143 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:20.800215 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:20.802203 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.803508 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.806050 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:20.806933 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:20.807029 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:20.813684 (+ 6655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:20.813705 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:20.813710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:20.813764 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:20.818532 (+ 4768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.829387 (+ 10855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.829420 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.829424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.829755 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:20.830751 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:20.830757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:20.830760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:20.831292 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:20.831496 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:20.833243 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0924 00:50:20.833629 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 00:50:20.834215 7118 ts_manager.cc:194] Re-registered known tserver with Master: 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755)
I20260924 00:50:20.834578 7730 ts_manager.cc:194] Registered new tserver with Master: 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755)
I20260924 00:50:20.835325 7118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:47527
I20260924 00:50:20.835558 8109 heartbeater.cc:347] Connected to a master server at 127.0.222.60:35737
I20260924 00:50:20.835742 8109 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:20.836601 8109 heartbeater.cc:511] Master 127.0.222.60:35737 requested a full tablet report, sending...
I20260924 00:50:20.838856 7923 ts_manager.cc:194] Registered new tserver with Master: 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755)
I20260924 00:50:20.925462 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 7318
I20260924 00:50:20.939105 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:40137
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:42943
--webserver_port=42051
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141,127.0.222.60:35737 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:21.306104 8121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:21.306520 8121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:21.306684 8121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:21.314371 8121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:21.314505 8121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:21.314677 8121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:21.324770 8121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:40137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=42051
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141,127.0.222.60:35737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:21.326236 8121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:21.328007 8121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:21.342623 8126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:21.342581 8129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:21.342540 8127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:21.343436 8121 server_base.cc:1034] running on GCE node
I20260924 00:50:21.344138 8121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:21.345258 8121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:21.346478 8121 hybrid_clock.cc:648] HybridClock initialized: now 1790211021346416 us; error 52 us; skew 500 ppm
I20260924 00:50:21.346822 8121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:21 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:21.350250 8121 init.cc:377] Logged in from keytab as oryx/127.0.222.2@KRBTEST.COM (short username oryx)
I20260924 00:50:21.351935 8121 webserver.cc:460] Webserver started at http://127.0.222.2:42051/ using document root <none> and password file <none>
I20260924 00:50:21.352640 8121 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:21.352767 8121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:21.358657 8121 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 00:50:21.362396 8136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:21.364305 8121 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260924 00:50:21.364511 8121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "e5ddd2e4fe40404b903b5adeda9646ae"
format_stamp: "Formatted at 2026-09-24 00:50:17 on dist-test-slave-rtvz"
server_key: "d6be69bc89ae9c06dbb037dcdbb6992d"
server_key_iv: "4198a849251d289f90e41f59e86856dd"
server_key_version: "encryptionkey@0"
I20260924 00:50:21.365651 8121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:21.387806 8121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:21.392462 8121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:21.392823 8121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:21.393934 8121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:21.395419 8121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:21.395516 8121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:21.395635 8121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:21.395709 8121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:21.426011 8121 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:40137
I20260924 00:50:21.426106 8250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:40137 every 8 connection(s)
I20260924 00:50:21.427498 8121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:21.431046 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 8121
I20260924 00:50:21.431332 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 7451
Sep 24 00:50:21 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.3@KRBTEST.COM
Sep 24 00:50:21 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
Sep 24 00:50:21 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:21.459509 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:21.432197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.0.222.3:33573)
0924 00:50:21.433691 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:21.433700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:21.433731 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:50:21.433810 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:21.433819 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:21.434077 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:21.434237 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:21.436431 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.438297 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.440479 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.441421 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.442228 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:21.448166 (+ 5938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:21.448187 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:21.448207 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:21.448265 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:21.454023 (+ 5758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:21.455315 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:21.455324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:21.455328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:21.455589 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:21.456237 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:21.456243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:21.456246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:21.456724 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:21.458316 (+ 1592us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:21.458516 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 00:50:21.458838 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1331,"thread_start_us":1184,"threads_started":1}
Sep 24 00:50:21 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
I20260924 00:50:21.461545 7575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:21.461612 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:21.430626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:58883 (local address 127.0.222.61:36141)
0924 00:50:21.431814 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:21.431820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:21.434330 (+ 2510us) server_negotiation.cc:407] Connection header received
0924 00:50:21.436257 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:21.436262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:21.436351 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:21.436465 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:21.438920 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.440125 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.443232 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.443858 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.443960 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:21.450150 (+ 6190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:21.450159 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:21.450164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:21.450210 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:21.454482 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:21.457323 (+ 2841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:21.457328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:21.457332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:21.457840 (+ 508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:21.458653 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:21.458659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:21.458662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:21.459093 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:21.459247 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:21.460805 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0924 00:50:21.461054 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064,"thread_start_us":966,"threads_started":1}
I20260924 00:50:21.463238 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:21.431019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.222.2:58883)
0924 00:50:21.434172 (+ 3153us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:21.434225 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:21.436006 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:21.436909 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:21.436921 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:21.437777 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:21.438678 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:21.438706 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.440391 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.440398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:21.443071 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:21.443085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.444290 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.444297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:21.444625 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:21.445450 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:21.445484 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:21.449796 (+ 4312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:21.454662 (+ 4866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:21.454722 (+ 60us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:21.454744 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:21.455865 (+ 1121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:21.458085 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:21.458090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:21.458095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:21.458472 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:21.459330 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:21.459339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:21.460002 (+ 663us) client_negotiation.cc:769] Sending connection context
0924 00:50:21.460663 (+ 661us) client_negotiation.cc:240] Negotiation successful
0924 00:50:21.461444 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2845,"thread_start_us":5273,"threads_started":1}
I20260924 00:50:21.466704 8253 heartbeater.cc:347] Connected to a master server at 127.0.222.61:36141
I20260924 00:50:21.467046 8253 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:21.468380 8253 heartbeater.cc:511] Master 127.0.222.61:36141 requested a full tablet report, sending...
I20260924 00:50:21.470139 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:21.430380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:35455 (local address 127.0.222.62:42943)
0924 00:50:21.431463 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:21.431469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:21.436788 (+ 5319us) server_negotiation.cc:407] Connection header received
0924 00:50:21.443108 (+ 6320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:21.443114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:21.443205 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:21.443309 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:21.444368 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.445486 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.450075 (+ 4589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.450802 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.450956 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:21.458070 (+ 7114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:21.458077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:21.458082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:21.458131 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:21.464483 (+ 6352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:21.465792 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:21.465799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:21.465803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:21.466067 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:21.468159 (+ 2092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:21.468164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:21.468167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:21.468681 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:21.468881 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:21.469316 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 00:50:21.469558 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":1454,"threads_started":1}
I20260924 00:50:21.470238 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:21.431016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:35455)
0924 00:50:21.436694 (+ 5678us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:21.436710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:21.442924 (+ 6214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:21.443339 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:21.443343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:21.443928 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:21.444201 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:21.444212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.445609 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.445614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:21.449798 (+ 4184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:21.449813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.451088 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.451097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:21.451454 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:21.452180 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:21.452204 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:21.456765 (+ 4561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:21.464795 (+ 8030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:21.464801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:21.464805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:21.465634 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:21.467835 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:21.467840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:21.467842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:21.468011 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:21.468831 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:21.468836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:21.469022 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 00:50:21.469179 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 00:50:21.469631 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5518,"mutex_wait_us":389,"thread_start_us":2341,"threads_started":1}
I20260924 00:50:21.470623 7730 ts_manager.cc:194] Registered new tserver with Master: e5ddd2e4fe40404b903b5adeda9646ae (127.0.222.2:40137)
I20260924 00:50:21.471530 8251 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42943
I20260924 00:50:21.471704 8251 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:21.471803 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:21.436999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:41103 (local address 127.0.222.60:35737)
0924 00:50:21.446568 (+ 9569us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:21.446574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:21.447583 (+ 1009us) server_negotiation.cc:407] Connection header received
0924 00:50:21.447880 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:21.447886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:21.447973 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:21.448051 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:21.451613 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.453068 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.455609 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.456535 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.456670 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:21.462968 (+ 6298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:21.462976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:21.462980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:21.463024 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:21.468108 (+ 5084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:21.469043 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:21.469050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:21.469053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:21.469316 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:21.469888 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:21.469894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:21.469896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:21.470328 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:21.470453 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:21.470997 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 00:50:21.471285 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9415,"thread_start_us":10666,"threads_started":1}
I20260924 00:50:21.472443 8251 heartbeater.cc:511] Master 127.0.222.62:42943 requested a full tablet report, sending...
I20260924 00:50:21.472970 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:21.436977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:35737 (local address 127.0.222.2:41103)
0924 00:50:21.447459 (+ 10482us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:21.447482 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:21.447721 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:21.448310 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:21.448314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:21.449535 (+ 1221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:21.451408 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:21.451425 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.453316 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.453321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:21.455430 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:21.455446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:21.457081 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:21.457087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:21.457349 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:21.458204 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:21.458223 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:21.462684 (+ 4461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:21.468253 (+ 5569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:21.468257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:21.468261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:21.468910 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:21.469462 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:21.469467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:21.469471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:21.469743 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:21.470581 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:21.470586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:21.470740 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 00:50:21.471984 (+ 1244us) client_negotiation.cc:240] Negotiation successful
0924 00:50:21.472340 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10288,"thread_start_us":13001,"threads_started":1}
I20260924 00:50:21.473703 7118 ts_manager.cc:194] Re-registered known tserver with Master: e5ddd2e4fe40404b903b5adeda9646ae (127.0.222.2:40137)
I20260924 00:50:21.474160 8252 heartbeater.cc:347] Connected to a master server at 127.0.222.60:35737
I20260924 00:50:21.474329 8252 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:21.475090 7118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:35455
I20260924 00:50:21.475138 8252 heartbeater.cc:511] Master 127.0.222.60:35737 requested a full tablet report, sending...
I20260924 00:50:21.476433 7923 ts_manager.cc:194] Registered new tserver with Master: e5ddd2e4fe40404b903b5adeda9646ae (127.0.222.2:40137)
I20260924 00:50:21.554046 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 7451
I20260924 00:50:21.567368 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:33573
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:42943
--webserver_port=46817
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141,127.0.222.60:35737 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:50:21.839013 8108 heartbeater.cc:503] Master 127.0.222.62:42943 was elected leader, sending a full tablet report...
W20260924 00:50:21.899628 8264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:21.900002 8264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:21.900141 8264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:21.907756 8264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:21.907893 8264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:21.908056 8264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:21.918025 8264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:33573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=46817
--tserver_master_addrs=127.0.222.62:42943,127.0.222.61:36141,127.0.222.60:35737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:21.919457 8264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:21.921051 8264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:21.937114 8270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:21.937026 8269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:21.936739 8272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:21.937270 8264 server_base.cc:1034] running on GCE node
I20260924 00:50:21.938269 8264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:21.939396 8264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:21.940644 8264 hybrid_clock.cc:648] HybridClock initialized: now 1790211021940544 us; error 54 us; skew 500 ppm
I20260924 00:50:21.941049 8264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:21 dist-test-slave-rtvz krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:21.944766 8264 init.cc:377] Logged in from keytab as oryx/127.0.222.3@KRBTEST.COM (short username oryx)
I20260924 00:50:21.946524 8264 webserver.cc:460] Webserver started at http://127.0.222.3:46817/ using document root <none> and password file <none>
I20260924 00:50:21.947273 8264 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:21.947414 8264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:21.953935 8264 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 00:50:21.957475 8279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:21.959151 8264 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 00:50:21.959344 8264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "c1d92fed7ae44e08b7a54f52c9c03978"
format_stamp: "Formatted at 2026-09-24 00:50:17 on dist-test-slave-rtvz"
server_key: "4bb8142c256abebd04f50d382f1698a0"
server_key_iv: "cd4336dd07e1b5f72d6929997496ab8f"
server_key_version: "encryptionkey@0"
I20260924 00:50:21.960386 8264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:21.977542 8264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:21.982004 8264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:21.982266 8264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:21.983251 8264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:21.984645 8264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:21.984844 8264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:21.984993 8264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:21.985073 8264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:22.017467 8264 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33573
I20260924 00:50:22.017526 8394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33573 every 8 connection(s)
I20260924 00:50:22.018993 8264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:22.027972 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 8264
Sep 24 00:50:22 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
Sep 24 00:50:22 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.60@KRBTEST.COM
I20260924 00:50:22.047695 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.022059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:58443 (local address 127.0.222.62:42943)
0924 00:50:22.022511 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.022517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.023671 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 00:50:22.026442 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.026448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.026550 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.026665 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.028859 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.030014 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.032671 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.033461 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.033616 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.039799 (+ 6183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.039808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.039814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.039865 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.043753 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.044778 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.044784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.044787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.044968 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.045501 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.045506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.045510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.045895 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.046045 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.047090 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.047328 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":378,"threads_started":1}
I20260924 00:50:22.049736 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.022642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:58443)
0924 00:50:22.023509 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:22.023554 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:22.026087 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:22.026955 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:22.026964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:22.027660 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:22.028581 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.028615 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.030178 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.030184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.032469 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.032491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.033732 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.033741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.034124 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.034998 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:22.035034 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:22.039429 (+ 4395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:22.043898 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:22.043908 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:22.043929 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:22.044612 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:22.045078 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:22.045083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:22.045087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:22.045376 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:22.046068 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:22.046078 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:22.046601 (+ 523us) client_negotiation.cc:769] Sending connection context
0924 00:50:22.046960 (+ 359us) client_negotiation.cc:240] Negotiation successful
0924 00:50:22.047616 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":357,"threads_started":1}
I20260924 00:50:22.051683 8395 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42943
I20260924 00:50:22.052168 8395 heartbeater.cc:464] Registering TS with master...
Sep 24 00:50:22 dist-test-slave-rtvz krb5kdc[7074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.61@KRBTEST.COM
I20260924 00:50:22.053823 8395 heartbeater.cc:511] Master 127.0.222.62:42943 requested a full tablet report, sending...
I20260924 00:50:22.056111 7118 ts_manager.cc:194] Re-registered known tserver with Master: c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573)
I20260924 00:50:22.057126 7118 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:58443
I20260924 00:50:22.060372 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.022643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:35737 (local address 127.0.222.3:41827)
0924 00:50:22.029505 (+ 6862us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:22.029525 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:22.029726 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:22.032980 (+ 3254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:22.032985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:22.034796 (+ 1811us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:22.037077 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.037092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.040401 (+ 3309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.040407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.041994 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.042019 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.043401 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.043409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.043613 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.045633 (+ 2020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:22.045654 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:22.048716 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:22.054231 (+ 5515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:22.054236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:22.054240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:22.055115 (+ 875us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:22.057871 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:22.057877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:22.057880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:22.058121 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:22.058951 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:22.058956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:22.059171 (+ 215us) client_negotiation.cc:769] Sending connection context
0924 00:50:22.059331 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 00:50:22.059718 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6689,"mutex_wait_us":37,"thread_start_us":6404,"threads_started":1}
I20260924 00:50:22.064608 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.022350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:41827 (local address 127.0.222.60:35737)
0924 00:50:22.029017 (+ 6667us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.029023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.029614 (+ 591us) server_negotiation.cc:407] Connection header received
0924 00:50:22.029892 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.029899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.030000 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.030081 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.037398 (+ 7317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.040171 (+ 2773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.042231 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.042909 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.043054 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.049077 (+ 6023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.049086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.049092 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.049135 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.054039 (+ 4904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.055368 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.055375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.055379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.057701 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.058319 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.058326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.058330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.058806 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.063428 (+ 4622us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.063650 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.064046 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6532,"thread_start_us":4259,"threads_started":1}
I20260924 00:50:22.065783 8396 heartbeater.cc:347] Connected to a master server at 127.0.222.60:35737
I20260924 00:50:22.066001 8396 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:22.066766 8396 heartbeater.cc:511] Master 127.0.222.60:35737 requested a full tablet report, sending...
I20260924 00:50:22.068331 7923 ts_manager.cc:194] Registered new tserver with Master: c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573)
I20260924 00:50:22.070421 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.033997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.222.3:34359)
0924 00:50:22.042251 (+ 8254us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:22.042273 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:22.042416 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:22.042765 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:22.042770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:22.043415 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:22.043784 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.043796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.045129 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.045134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.046568 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.046582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.048522 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.048532 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.048787 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.049539 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:22.049561 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:22.053735 (+ 4174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:22.066907 (+ 13172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:22.066912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:22.066916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:22.067602 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:22.068098 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:22.068104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:22.068108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:22.068319 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:22.069074 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:22.069079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:22.069306 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 00:50:22.069460 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 00:50:22.069839 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8107,"thread_start_us":9873,"threads_started":1}
I20260924 00:50:22.071529 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.029859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:34359 (local address 127.0.222.61:36141)
0924 00:50:22.041963 (+ 12104us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.041969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.042446 (+ 477us) server_negotiation.cc:407] Connection header received
0924 00:50:22.042510 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.042515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.042599 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.042710 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.043917 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.045018 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.046772 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.048213 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.048359 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.054026 (+ 5667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.054036 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.054042 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.054153 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.066754 (+ 12601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.067720 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.067728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.067732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.067993 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.068453 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.068459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.068463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.068924 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.069111 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.070202 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.070915 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11942,"thread_start_us":11787,"threads_started":1}
I20260924 00:50:22.072372 8397 heartbeater.cc:347] Connected to a master server at 127.0.222.61:36141
I20260924 00:50:22.072551 8397 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:22.073109 8397 heartbeater.cc:511] Master 127.0.222.61:36141 requested a full tablet report, sending...
I20260924 00:50:22.074577 7730 ts_manager.cc:194] Registered new tserver with Master: c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573)
I20260924 00:50:22.091260 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.057646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.0.222.62:42943)
0924 00:50:22.058005 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.058010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.069512 (+ 11502us) server_negotiation.cc:407] Connection header received
0924 00:50:22.069904 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.069911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.070035 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.070112 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.074043 (+ 3931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.075332 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.078004 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.078837 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.078975 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.083748 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.083757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.083763 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.083818 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.087689 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.088614 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.088620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.088623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.088827 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.089354 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.089359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.089363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.089852 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.090176 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.090526 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.090771 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260924 00:50:22.096623 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.069876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.222.60:35737)
0924 00:50:22.070050 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.070056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.076329 (+ 6273us) server_negotiation.cc:407] Connection header received
0924 00:50:22.076638 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.076644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.076729 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.076804 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.080573 (+ 3769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.081940 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.084099 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.084750 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.084861 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.088873 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.088880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.088883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.088924 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.092921 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.093953 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.093959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.093963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.094180 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.094675 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.094680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.094683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.095151 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.095411 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.095858 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.096135 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:50:22.098203 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.057804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.0.222.61:36141)
0924 00:50:22.065967 (+ 8163us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.065974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.066584 (+ 610us) server_negotiation.cc:407] Connection header received
0924 00:50:22.066654 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.066660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.066760 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.066937 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.077687 (+ 10750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.079212 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.081767 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.082805 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.082943 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.088797 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.088806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.088812 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.088881 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.093488 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.094738 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.094744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.094748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.094944 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.096191 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.096197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.096200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.096549 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.096798 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.097440 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.097693 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7911,"thread_start_us":9076,"threads_started":1}
I20260924 00:50:22.096916 7118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:50:22.108314 7118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:22.148833 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.133658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33573 (local address 127.0.0.1:47762)
0924 00:50:22.134800 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:22.134823 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:22.134975 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:22.137915 (+ 2940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:22.137920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:22.137941 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:22.138572 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.138584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.140646 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.140653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.144502 (+ 3849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.144521 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.147674 (+ 3153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.147682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.147932 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.147973 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 00:50:22.148074 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 00:50:22.148234 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":681,"threads_started":1}
I20260924 00:50:22.148890 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.134164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.0.222.3:33573)
0924 00:50:22.137145 (+ 2981us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.137154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.137223 (+ 69us) server_negotiation.cc:407] Connection header received
0924 00:50:22.137347 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.137356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.137649 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.137927 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:22.138801 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.140503 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.144700 (+ 4197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.147224 (+ 2524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.147344 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.147635 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.148136 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.148349 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2770,"thread_start_us":2543,"threads_started":1}
I20260924 00:50:22.151033 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.134412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:40137 (local address 127.0.0.1:37652)
0924 00:50:22.135344 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:22.135364 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:22.135497 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:22.141725 (+ 6228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:22.141735 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:22.141767 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:22.142866 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.142884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.145003 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.145009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.147428 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.147447 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.149864 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.149872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.150194 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.150222 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 00:50:22.150327 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 00:50:22.150493 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":797,"threads_started":1}
I20260924 00:50:22.152400 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.135012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.0.222.2:40137)
0924 00:50:22.135743 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.135751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.135795 (+ 44us) server_negotiation.cc:407] Connection header received
0924 00:50:22.135935 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.135944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.141460 (+ 5516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.141711 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:22.143146 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.144827 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.147622 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.149553 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.149677 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.149942 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.150440 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.152014 (+ 1574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":441,"threads_started":1}
I20260924 00:50:22.154354 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.136838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.0.222.1:46755)
0924 00:50:22.138201 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.138212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.143393 (+ 5181us) server_negotiation.cc:407] Connection header received
0924 00:50:22.143584 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.143591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.143802 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.144012 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:22.145162 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.147095 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.150494 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.152527 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.152632 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.152867 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.153704 (+ 837us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.153912 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":495,"threads_started":1}
I20260924 00:50:22.154489 8329 tablet_service.cc:1467] Processing CreateTablet for tablet 186d5f9d991a4a3b81f3193e53a57966 (DEFAULT_TABLE table=test-table [id=e4b62d1c95024d758e76419061a14dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:22.155396 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.136512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:46755 (local address 127.0.0.1:37346)
0924 00:50:22.143264 (+ 6752us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:22.143280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:22.143410 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:22.144003 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:22.144007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:22.144033 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:22.144885 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.144899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.147341 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.147346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.150309 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.150325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.153164 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.153172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.153458 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.153493 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:50:22.154636 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0924 00:50:22.154805 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6597,"mutex_wait_us":1,"thread_start_us":5961,"threads_started":1}
I20260924 00:50:22.156323 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186d5f9d991a4a3b81f3193e53a57966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:22.157583 8185 tablet_service.cc:1467] Processing CreateTablet for tablet 186d5f9d991a4a3b81f3193e53a57966 (DEFAULT_TABLE table=test-table [id=e4b62d1c95024d758e76419061a14dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:22.158951 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186d5f9d991a4a3b81f3193e53a57966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:22.160346 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 186d5f9d991a4a3b81f3193e53a57966 (DEFAULT_TABLE table=test-table [id=e4b62d1c95024d758e76419061a14dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:22.162110 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186d5f9d991a4a3b81f3193e53a57966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:22.180985 8430 tablet_bootstrap.cc:492] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Bootstrap starting.
I20260924 00:50:22.187983 8429 tablet_bootstrap.cc:492] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Bootstrap starting.
I20260924 00:50:22.190927 8431 tablet_bootstrap.cc:492] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Bootstrap starting.
I20260924 00:50:22.191047 8430 tablet_bootstrap.cc:654] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:22.192888 8430 log.cc:824] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:22.193619 8429 tablet_bootstrap.cc:654] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:22.195688 8429 log.cc:824] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:22.196534 8431 tablet_bootstrap.cc:654] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:22.197154 8430 tablet_bootstrap.cc:492] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: No bootstrap required, opened a new log
I20260924 00:50:22.197582 8430 ts_tablet_manager.cc:1397] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260924 00:50:22.198554 8431 log.cc:824] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:22.200016 8429 tablet_bootstrap.cc:492] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: No bootstrap required, opened a new log
I20260924 00:50:22.200349 8429 ts_tablet_manager.cc:1397] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260924 00:50:22.202817 8431 tablet_bootstrap.cc:492] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: No bootstrap required, opened a new log
I20260924 00:50:22.203277 8431 ts_tablet_manager.cc:1397] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260924 00:50:22.210104 8430 raft_consensus.cc:348] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.210656 8430 raft_consensus.cc:374] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:22.210811 8430 raft_consensus.cc:729] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d92fed7ae44e08b7a54f52c9c03978, State: Initialized, Role: FOLLOWER
I20260924 00:50:22.211539 8430 consensus_queue.cc:260] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.213325 8395 heartbeater.cc:503] Master 127.0.222.62:42943 was elected leader, sending a full tablet report...
I20260924 00:50:22.214207 8430 ts_tablet_manager.cc:1428] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260924 00:50:22.216928 8429 raft_consensus.cc:348] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.217589 8429 raft_consensus.cc:374] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:22.217744 8429 raft_consensus.cc:729] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ddd2e4fe40404b903b5adeda9646ae, State: Initialized, Role: FOLLOWER
I20260924 00:50:22.218567 8429 consensus_queue.cc:260] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.220583 8251 heartbeater.cc:503] Master 127.0.222.62:42943 was elected leader, sending a full tablet report...
I20260924 00:50:22.221591 8429 ts_tablet_manager.cc:1428] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Time spent starting tablet: real 0.021s user 0.015s sys 0.006s
I20260924 00:50:22.223187 8431 raft_consensus.cc:348] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.224639 8431 raft_consensus.cc:374] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:22.224834 8431 raft_consensus.cc:729] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3359189faab547c5a64c98e926d18930, State: Initialized, Role: FOLLOWER
I20260924 00:50:22.226249 8431 consensus_queue.cc:260] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.228588 8431 ts_tablet_manager.cc:1428] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Time spent starting tablet: real 0.025s user 0.004s sys 0.013s
W20260924 00:50:22.267830 8111 tablet.cc:2369] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:50:22.271492 8398 tablet.cc:2369] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:50:22.362816 8436 raft_consensus.cc:482] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:22.363322 8436 raft_consensus.cc:504] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.366267 8436 leader_election.cc:288] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573), 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755)
I20260924 00:50:22.375808 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.366695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:48693 (local address 127.0.222.3:33573)
0924 00:50:22.367014 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.367021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.367990 (+ 969us) server_negotiation.cc:407] Connection header received
0924 00:50:22.368311 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.368318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.368404 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.368504 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:22.369600 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.370787 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.373609 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.374748 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.374871 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.375010 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.375395 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.375521 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260924 00:50:22.376020 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.367022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:40077 (local address 127.0.222.1:46755)
0924 00:50:22.367275 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.367280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.368453 (+ 1173us) server_negotiation.cc:407] Connection header received
0924 00:50:22.368638 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.368644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.368769 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.368870 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:22.370077 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.371112 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.373709 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.374911 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.375031 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.375134 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.375551 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.375680 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 00:50:22.376324 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.366413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:46755 (local address 127.0.222.2:40077)
0924 00:50:22.368345 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:22.368362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:22.368516 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:22.368898 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:22.368901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:22.368921 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:22.369855 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.369867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.371292 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.371296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.373553 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.373567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.375221 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.375228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.375395 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.375418 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 00:50:22.375500 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:50:22.375632 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1709,"mutex_wait_us":688,"thread_start_us":733,"threads_started":1}
I20260924 00:50:22.376457 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.366367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33573 (local address 127.0.222.2:48693)
0924 00:50:22.367828 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:22.367851 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:22.368056 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:22.368548 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:22.368554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:22.368582 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:22.369408 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.369428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.370951 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.370956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.373418 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:22.373432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.375020 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.375028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:22.375233 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.375260 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 00:50:22.375349 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:50:22.375492 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"mutex_wait_us":2,"thread_start_us":748,"threads_started":1}
I20260924 00:50:22.377708 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186d5f9d991a4a3b81f3193e53a57966" candidate_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3359189faab547c5a64c98e926d18930" is_pre_election: true
I20260924 00:50:22.377698 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186d5f9d991a4a3b81f3193e53a57966" candidate_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" is_pre_election: true
I20260924 00:50:22.378413 8349 raft_consensus.cc:2454] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5ddd2e4fe40404b903b5adeda9646ae in term 0.
I20260924 00:50:22.378400 8061 raft_consensus.cc:2454] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5ddd2e4fe40404b903b5adeda9646ae in term 0.
I20260924 00:50:22.379256 8139 leader_election.cc:302] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1d92fed7ae44e08b7a54f52c9c03978, e5ddd2e4fe40404b903b5adeda9646ae; no voters:
I20260924 00:50:22.379745 8436 raft_consensus.cc:2781] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:22.379959 8436 raft_consensus.cc:482] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:22.380095 8436 raft_consensus.cc:3037] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:22.383379 8436 raft_consensus.cc:504] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.384173 8436 leader_election.cc:288] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [CANDIDATE]: Term 1 election: Requested vote from peers c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573), 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755)
I20260924 00:50:22.384799 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186d5f9d991a4a3b81f3193e53a57966" candidate_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d92fed7ae44e08b7a54f52c9c03978"
I20260924 00:50:22.384806 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186d5f9d991a4a3b81f3193e53a57966" candidate_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3359189faab547c5a64c98e926d18930"
I20260924 00:50:22.385051 8061 raft_consensus.cc:3037] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:22.385058 8349 raft_consensus.cc:3037] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:22.388303 8349 raft_consensus.cc:2454] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5ddd2e4fe40404b903b5adeda9646ae in term 1.
I20260924 00:50:22.388355 8061 raft_consensus.cc:2454] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5ddd2e4fe40404b903b5adeda9646ae in term 1.
I20260924 00:50:22.388921 8137 leader_election.cc:302] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3359189faab547c5a64c98e926d18930, e5ddd2e4fe40404b903b5adeda9646ae; no voters:
I20260924 00:50:22.389389 8436 raft_consensus.cc:2781] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:22.390161 8436 raft_consensus.cc:686] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 LEADER]: Becoming Leader. State: Replica: e5ddd2e4fe40404b903b5adeda9646ae, State: Running, Role: LEADER
I20260924 00:50:22.390826 8436 consensus_queue.cc:237] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } }
I20260924 00:50:22.397014 7118 catalog_manager.cc:5495] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae reported cstate change: term changed from 0 to 1, leader changed from <none> to e5ddd2e4fe40404b903b5adeda9646ae (127.0.222.2). New cstate: current_term: 1 leader_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } health_report { overall_health: UNKNOWN } } }
W20260924 00:50:22.431095 8254 tablet.cc:2369] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:50:22.470450 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.463907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.222.2:40137)
0924 00:50:22.464091 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.464095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.464113 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:50:22.464286 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.464291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.464387 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.464514 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:22.465011 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.465923 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.467656 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.468198 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.468294 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.468999 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:22.469751 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:22.469894 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.469999 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.470071 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:50:22.480883 8061 raft_consensus.cc:1261] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 1 FOLLOWER]: Refusing update from remote peer e5ddd2e4fe40404b903b5adeda9646ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:22.480906 8349 raft_consensus.cc:1261] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 1 FOLLOWER]: Refusing update from remote peer e5ddd2e4fe40404b903b5adeda9646ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:22.482072 8436 consensus_queue.cc:1035] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:22.482620 8440 consensus_queue.cc:1035] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:22.499110 8444 mvcc.cc:205] Tried to move back new op lower bound from 7332704348070498304 to 7332704347717812224. Current Snapshot: MvccSnapshot[applied={T|T < 7332704348070498304}]
I20260924 00:50:22.502028 8446 mvcc.cc:205] Tried to move back new op lower bound from 7332704348070498304 to 7332704347717812224. Current Snapshot: MvccSnapshot[applied={T|T < 7332704348070498304}]
I20260924 00:50:22.521760 7119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:22.522130 7119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:22.531764 7119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bff8d21582e43049659eaed39fe3937: Sending DeleteTablet for 3 replicas of tablet 186d5f9d991a4a3b81f3193e53a57966
I20260924 00:50:22.533318 8329 tablet_service.cc:1514] Processing DeleteTablet for tablet 186d5f9d991a4a3b81f3193e53a57966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:22 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:47762
I20260924 00:50:22.533483 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 186d5f9d991a4a3b81f3193e53a57966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:22 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:37346
I20260924 00:50:22.534353 8185 tablet_service.cc:1514] Processing DeleteTablet for tablet 186d5f9d991a4a3b81f3193e53a57966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:22 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:37652
I20260924 00:50:22.535763 8456 tablet_replica.cc:330] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: stopping tablet replica
I20260924 00:50:22.536396 8456 raft_consensus.cc:2229] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:50:22.537552 8456 raft_consensus.cc:2258] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:22.544087 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 7087
I20260924 00:50:22.544162 8454 tablet_replica.cc:330] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: stopping tablet replica
I20260924 00:50:22.544759 8454 raft_consensus.cc:2229] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:22.545431 7150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:22.545629 8454 raft_consensus.cc:2258] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:22.547178 8455 tablet_replica.cc:330] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: stopping tablet replica
I20260924 00:50:22.547756 8455 raft_consensus.cc:2229] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:22.548559 8455 raft_consensus.cc:2258] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:22.549183 8456 ts_tablet_manager.cc:1905] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:22.553525 8455 ts_tablet_manager.cc:1905] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:22.557112 8454 ts_tablet_manager.cc:1905] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:22.560774 8456 ts_tablet_manager.cc:1918] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:22.561064 8456 log.cc:1196] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/186d5f9d991a4a3b81f3193e53a57966
I20260924 00:50:22.561888 8456 ts_tablet_manager.cc:1939] T 186d5f9d991a4a3b81f3193e53a57966 P e5ddd2e4fe40404b903b5adeda9646ae: Deleting consensus metadata
I20260924 00:50:22.564082 8455 ts_tablet_manager.cc:1918] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:22.564332 8455 log.cc:1196] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/186d5f9d991a4a3b81f3193e53a57966
I20260924 00:50:22.565184 8455 ts_tablet_manager.cc:1939] T 186d5f9d991a4a3b81f3193e53a57966 P 3359189faab547c5a64c98e926d18930: Deleting consensus metadata
I20260924 00:50:22.572791 8454 ts_tablet_manager.cc:1918] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:22.573084 8454 log.cc:1196] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/186d5f9d991a4a3b81f3193e53a57966
I20260924 00:50:22.573944 8454 ts_tablet_manager.cc:1939] T 186d5f9d991a4a3b81f3193e53a57966 P c1d92fed7ae44e08b7a54f52c9c03978: Deleting consensus metadata
I20260924 00:50:22.576191 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.538064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.0.222.61:36141)
0924 00:50:22.538304 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.538310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.542441 (+ 4131us) server_negotiation.cc:407] Connection header received
0924 00:50:22.544011 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.544018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.544117 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.544221 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.549593 (+ 5372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.550617 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.554478 (+ 3861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.555526 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.555648 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.566100 (+ 10452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.566110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.566116 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.566178 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.571279 (+ 5101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.572750 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.572756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.572760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.572938 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.573558 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.573563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.573566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.573949 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.574191 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.575519 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.575781 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:50:22.593627 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.542762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.0.222.60:35737)
0924 00:50:22.542936 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.542942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.576205 (+ 33263us) server_negotiation.cc:407] Connection header received
0924 00:50:22.576474 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.576480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.576564 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.576634 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.579074 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.580086 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.581916 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.582687 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.582809 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.586289 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.586295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.586298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.586336 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.589877 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.590705 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.590711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.590715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.590873 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.591425 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.591431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.591434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.591827 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.592186 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.592730 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.593044 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:50:22.714120 7106 catalog_manager.cc:4841] TS e5ddd2e4fe40404b903b5adeda9646ae (127.0.222.2:40137): tablet 186d5f9d991a4a3b81f3193e53a57966 (table test-table [id=e4b62d1c95024d758e76419061a14dc2]) successfully deleted
I20260924 00:50:22.714663 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 7087
I20260924 00:50:22.756418 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.740062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.0.222.60:35737)
0924 00:50:22.740246 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.740252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.740272 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:50:22.740815 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.740821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.740911 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.741013 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.743074 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.743864 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.746027 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.746487 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.746606 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.750471 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.750477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.750480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.750517 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.753235 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.753974 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.753979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.753981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.754134 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.754641 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.754647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.754649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.754993 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.755257 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.755658 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.755913 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:50:22.757875 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:22.738242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39444 (local address 127.0.222.61:36141)
0924 00:50:22.738447 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:22.738453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:22.740095 (+ 1642us) server_negotiation.cc:407] Connection header received
0924 00:50:22.740299 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:22.740305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:22.740391 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:22.740474 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:22.742600 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.743904 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.745995 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:22.746650 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:22.746779 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:22.750448 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:22.750455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:22.750460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:22.750507 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:22.753940 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.754725 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.754731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.754734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.754896 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:22.755405 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:22.755409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:22.755411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:22.755800 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:22.756031 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:22.757285 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0924 00:50:22.757529 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:50:23.505545 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:23.504739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:56789)
0924 00:50:23.505188 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:23.505357 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":208,"threads_started":1}
W20260924 00:50:23.506031 8395 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42943 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:23.515247 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:23.514463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:41699)
0924 00:50:23.514929 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:23.515043 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
W20260924 00:50:23.515904 8108 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42943 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:23.516592 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:23.515534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:52947)
0924 00:50:23.516271 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:23.516404 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
W20260924 00:50:23.516985 8251 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42943 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:24.030700 8468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1bff8d21582e43049659eaed39fe3937)
I20260924 00:50:24.031136 8469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1bff8d21582e43049659eaed39fe3937)
I20260924 00:50:24.031070 8468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } }
I20260924 00:50:24.031376 8469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } }
I20260924 00:50:24.032538 8468 leader_election.cc:288] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943), 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61:36141)
I20260924 00:50:24.032647 8469 leader_election.cc:288] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943), 3986ed3b261f467fa0d79609b31b073c (127.0.222.60:35737)
I20260924 00:50:24.048415 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.032743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46648)
0924 00:50:24.047922 (+ 15179us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.048117 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15052,"threads_started":1}
W20260924 00:50:24.049103 7909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:24.051244 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.033126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46658)
0924 00:50:24.050760 (+ 17634us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.050927 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17513,"thread_start_us":17383,"threads_started":1}
I20260924 00:50:24.051376 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.050842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46680)
0924 00:50:24.051132 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.051228 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
W20260924 00:50:24.051539 7909 leader_election.cc:334] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943): Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
W20260924 00:50:24.051939 7716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:24.053843 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.053394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46688)
0924 00:50:24.053608 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.053698 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260924 00:50:24.054118 7716 leader_election.cc:334] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943): Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:24.057772 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.050764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.0.222.61:36141)
0924 00:50:24.051957 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:24.051961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:24.051976 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:50:24.052026 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:24.052029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:24.052094 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:24.052192 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:24.053105 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:24.054258 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:24.055949 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:24.056723 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:24.056844 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:24.056973 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:24.057277 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 00:50:24.057413 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":862,"threads_started":1}
I20260924 00:50:24.057829 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.049296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.0.222.60:35737)
0924 00:50:24.050566 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:24.050571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:24.050594 (+ 23us) server_negotiation.cc:407] Connection header received
0924 00:50:24.051044 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:24.051050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:24.051133 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:24.051239 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:24.052494 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:24.053780 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:24.055960 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:24.056799 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:24.056923 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:24.057037 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:24.057311 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 00:50:24.057446 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1144,"thread_start_us":1006,"threads_started":1}
I20260924 00:50:24.057821 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.033076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:35737 (local address 127.0.0.1:56870)
0924 00:50:24.049379 (+ 16303us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.049399 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:24.049538 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:24.051862 (+ 2324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:24.051867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:24.051899 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:24.052234 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:24.052248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:24.053940 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:24.053946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:24.055801 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:24.055815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:24.056999 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:24.057006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:24.057153 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:24.057179 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 00:50:24.057291 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 00:50:24.057440 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16161,"thread_start_us":16541,"threads_started":1}
I20260924 00:50:24.057850 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.032799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:36141 (local address 127.0.0.1:39456)
0924 00:50:24.048731 (+ 15932us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.048754 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:24.048915 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:24.052312 (+ 3397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:24.052318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:24.052346 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:24.052873 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:24.052888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:24.054421 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:24.054425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:24.055788 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:24.055802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:24.056911 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:24.056917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:24.057103 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:24.057127 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 00:50:24.057233 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 00:50:24.057376 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15792,"mutex_wait_us":56,"thread_start_us":16202,"threads_started":1}
I20260924 00:50:24.058791 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3986ed3b261f467fa0d79609b31b073c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e488b41d2c04decbea7acdaa202c91e" is_pre_election: true
I20260924 00:50:24.058763 7933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e488b41d2c04decbea7acdaa202c91e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3986ed3b261f467fa0d79609b31b073c" is_pre_election: true
I20260924 00:50:24.059093 7933 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e488b41d2c04decbea7acdaa202c91e in term 1.
I20260924 00:50:24.059082 7741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3986ed3b261f467fa0d79609b31b073c in term 1.
I20260924 00:50:24.059597 7910 leader_election.cc:302] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3986ed3b261f467fa0d79609b31b073c, 6e488b41d2c04decbea7acdaa202c91e; no voters: 1bff8d21582e43049659eaed39fe3937
I20260924 00:50:24.059595 7717 leader_election.cc:302] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3986ed3b261f467fa0d79609b31b073c, 6e488b41d2c04decbea7acdaa202c91e; no voters: 1bff8d21582e43049659eaed39fe3937
I20260924 00:50:24.059970 8468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:50:24.059970 8469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:50:24.060098 8469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1bff8d21582e43049659eaed39fe3937)
I20260924 00:50:24.060099 8468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1bff8d21582e43049659eaed39fe3937)
I20260924 00:50:24.060187 8469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:50:24.060201 8468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:50:24.062383 8469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } }
I20260924 00:50:24.062382 8468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } }
I20260924 00:50:24.063074 8468 leader_election.cc:288] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 2 election: Requested vote from peers 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943), 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61:36141)
I20260924 00:50:24.063189 8469 leader_election.cc:288] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [CANDIDATE]: Term 2 election: Requested vote from peers 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943), 3986ed3b261f467fa0d79609b31b073c (127.0.222.60:35737)
I20260924 00:50:24.063699 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.063206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46692)
0924 00:50:24.063445 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.063536 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 00:50:24.063716 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.063220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46708)
0924 00:50:24.063455 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.063542 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 00:50:24.063894 7933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e488b41d2c04decbea7acdaa202c91e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3986ed3b261f467fa0d79609b31b073c"
I20260924 00:50:24.064003 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3986ed3b261f467fa0d79609b31b073c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e488b41d2c04decbea7acdaa202c91e"
I20260924 00:50:24.064242 7933 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e488b41d2c04decbea7acdaa202c91e in current term 2: Already voted for candidate 3986ed3b261f467fa0d79609b31b073c in this term.
I20260924 00:50:24.064270 7741 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3986ed3b261f467fa0d79609b31b073c in current term 2: Already voted for candidate 6e488b41d2c04decbea7acdaa202c91e in this term.
I20260924 00:50:24.065084 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.064551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46724)
0924 00:50:24.064835 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.064926 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260924 00:50:24.065253 7909 leader_election.cc:334] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943): Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:24.065400 7909 leader_election.cc:302] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3986ed3b261f467fa0d79609b31b073c; no voters: 1bff8d21582e43049659eaed39fe3937, 6e488b41d2c04decbea7acdaa202c91e
I20260924 00:50:24.065467 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.065022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46734)
0924 00:50:24.065248 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.065332 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 00:50:24.065632 7716 leader_election.cc:334] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943): Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:24.065768 7716 leader_election.cc:302] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e488b41d2c04decbea7acdaa202c91e; no voters: 1bff8d21582e43049659eaed39fe3937, 3986ed3b261f467fa0d79609b31b073c
I20260924 00:50:24.065941 8468 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:50:24.066012 8469 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 00:50:24.507617 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.506838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:39697)
0924 00:50:24.507343 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.507442 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
I20260924 00:50:24.517663 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.516676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:49875)
0924 00:50:24.517299 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.517416 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
I20260924 00:50:24.519249 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:24.517623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:53831)
0924 00:50:24.518985 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:24.519092 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":1133,"threads_started":1}
I20260924 00:50:25.509536 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.508713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:46875)
0924 00:50:25.509190 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.509349 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260924 00:50:25.519600 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.518619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:56981)
0924 00:50:25.519226 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.519365 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":271,"threads_started":1}
I20260924 00:50:25.521138 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.520357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:56663)
0924 00:50:25.520880 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.520964 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":324,"threads_started":1}
I20260924 00:50:25.777508 8484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:25.777829 8484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } }
I20260924 00:50:25.778522 8484 leader_election.cc:288] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943), 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61:36141)
I20260924 00:50:25.779220 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3986ed3b261f467fa0d79609b31b073c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e488b41d2c04decbea7acdaa202c91e" is_pre_election: true
I20260924 00:50:25.779493 7741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3986ed3b261f467fa0d79609b31b073c in term 2.
I20260924 00:50:25.779599 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.778694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46972)
0924 00:50:25.779235 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.779365 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":248,"threads_started":1}
I20260924 00:50:25.780025 7910 leader_election.cc:302] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3986ed3b261f467fa0d79609b31b073c, 6e488b41d2c04decbea7acdaa202c91e; no voters:
I20260924 00:50:25.780364 8484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 00:50:25.780500 8484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:25.780596 8484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:50:25.781062 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.780677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:46988)
0924 00:50:25.780810 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.780922 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 00:50:25.781266 7909 leader_election.cc:334] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943): Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:25.782886 8484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } }
I20260924 00:50:25.783460 8484 leader_election.cc:288] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 3 election: Requested vote from peers 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943), 6e488b41d2c04decbea7acdaa202c91e (127.0.222.61:36141)
I20260924 00:50:25.784027 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.783630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:47000)
0924 00:50:25.783773 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.783875 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 00:50:25.783973 7741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3986ed3b261f467fa0d79609b31b073c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e488b41d2c04decbea7acdaa202c91e"
I20260924 00:50:25.784221 7741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:50:25.784971 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.784599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:47004)
0924 00:50:25.784751 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.784834 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 00:50:25.785138 7909 leader_election.cc:334] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943): Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
I20260924 00:50:25.786574 7741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3986ed3b261f467fa0d79609b31b073c in term 3.
I20260924 00:50:25.787005 7910 leader_election.cc:302] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3986ed3b261f467fa0d79609b31b073c, 6e488b41d2c04decbea7acdaa202c91e; no voters: 1bff8d21582e43049659eaed39fe3937
I20260924 00:50:25.787281 8484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:50:25.787698 8484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [term 3 LEADER]: Becoming Leader. State: Replica: 3986ed3b261f467fa0d79609b31b073c, State: Running, Role: LEADER
I20260924 00:50:25.788080 8484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } }
I20260924 00:50:25.789717 8488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3986ed3b261f467fa0d79609b31b073c. Latest consensus state: current_term: 3 leader_uuid: "3986ed3b261f467fa0d79609b31b073c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:25.790084 8488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:25.790758 8490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:25.794353 8490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:25.795184 8490 catalog_manager.cc:1259] Loaded cluster ID: 7981b32f7ae2469a905d7dc221204823
I20260924 00:50:25.795254 8490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:25.796391 8490 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:25.797243 8490 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Loaded TSK: 0
I20260924 00:50:25.798439 8490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:50:25.883872 7922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:50:25.890941 7922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:25.895262 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.894841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:47028)
0924 00:50:25.895005 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.895102 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 00:50:25.895401 7741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [term 3 FOLLOWER]: Refusing update from remote peer 3986ed3b261f467fa0d79609b31b073c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 00:50:25.896090 8488 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 00:50:25.896415 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.896043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:47032)
0924 00:50:25.896207 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.896276 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 00:50:25.896601 7909 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c -> Peer 1bff8d21582e43049659eaed39fe3937 (127.0.222.62:42943): Couldn't send request to peer 1bff8d21582e43049659eaed39fe3937. Status: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:50:25.898478 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3986ed3b261f467fa0d79609b31b073c. Latest consensus state: current_term: 3 leader_uuid: "3986ed3b261f467fa0d79609b31b073c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:25.898742 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:50:25.900490 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3986ed3b261f467fa0d79609b31b073c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:25.900822 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:25.901015 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3986ed3b261f467fa0d79609b31b073c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1bff8d21582e43049659eaed39fe3937" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42943 } } peers { permanent_uuid: "6e488b41d2c04decbea7acdaa202c91e" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 36141 } attrs { promote: false } } peers { permanent_uuid: "3986ed3b261f467fa0d79609b31b073c" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 35737 } attrs { promote: false } } }
I20260924 00:50:25.901307 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e488b41d2c04decbea7acdaa202c91e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:50:25.918236 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.909292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:46755 (local address 127.0.0.1:37360)
0924 00:50:25.909498 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.909517 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:25.909658 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:25.910594 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:25.910601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:25.910630 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:25.911029 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:25.911043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.912736 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.912744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:25.915225 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:25.915250 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.917251 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.917260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:25.917488 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:25.917539 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 00:50:25.917626 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:50:25.917808 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 00:50:25.919687 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.909667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.0.222.1:46755)
0924 00:50:25.910223 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:25.910229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:25.910248 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:50:25.910326 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:25.910332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:25.910427 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:25.911185 (+ 758us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:25.911394 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.912557 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.915535 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.916959 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.918875 (+ 1916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:25.919017 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:25.919087 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 00:50:25.919273 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":522,"threads_started":1}
I20260924 00:50:25.920830 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 62bd76bc4b6f43e5b85b865669ce0752 (DEFAULT_TABLE table=test-table [id=ab668f18daac434cad21d08b7c27abef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:25.921382 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62bd76bc4b6f43e5b85b865669ce0752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:25.929148 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.909697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33573 (local address 127.0.0.1:47772)
0924 00:50:25.915944 (+ 6247us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.915971 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:25.916145 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:25.922200 (+ 6055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:25.922208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:25.922245 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:25.922653 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:25.922667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.924166 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.924171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:25.926154 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:25.926174 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.928220 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.928228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:25.928459 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:25.928503 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 00:50:25.928599 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:50:25.928778 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6102,"thread_start_us":11494,"threads_started":1}
I20260924 00:50:25.929924 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.910074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.0.222.3:33573)
0924 00:50:25.921814 (+ 11740us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:25.921821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:25.921842 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:50:25.921917 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:25.921922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:25.922014 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:25.922144 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:25.922915 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.923989 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.926372 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.927771 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.927906 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:25.928017 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:25.929434 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0924 00:50:25.929601 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11605,"thread_start_us":12254,"threads_started":1}
I20260924 00:50:25.931036 8329 tablet_service.cc:1467] Processing CreateTablet for tablet 62bd76bc4b6f43e5b85b865669ce0752 (DEFAULT_TABLE table=test-table [id=ab668f18daac434cad21d08b7c27abef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:25.931557 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62bd76bc4b6f43e5b85b865669ce0752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:25.933784 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.912945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:40137 (local address 127.0.0.1:37676)
0924 00:50:25.918530 (+ 5585us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.918550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:25.918662 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:25.927902 (+ 9240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:25.927908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:25.927942 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:25.928130 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:25.928145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.929690 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.929696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:25.931352 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:25.931370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.932959 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.932965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:25.933168 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:25.933237 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 00:50:25.933315 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 00:50:25.933483 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5478,"thread_start_us":2694,"threads_started":1}
I20260924 00:50:25.934118 8504 tablet_bootstrap.cc:492] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: Bootstrap starting.
I20260924 00:50:25.934782 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.913550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.0.222.2:40137)
0924 00:50:25.927566 (+ 14016us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:25.927573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:25.927598 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:50:25.927673 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:25.927679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:25.927778 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:25.927884 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:25.928315 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.929520 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.931615 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:25.932761 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:25.934142 (+ 1381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:25.934251 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:25.934313 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 00:50:25.934458 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13855,"thread_start_us":14476,"threads_started":1}
I20260924 00:50:25.935638 8504 tablet_bootstrap.cc:654] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:25.935556 8185 tablet_service.cc:1467] Processing CreateTablet for tablet 62bd76bc4b6f43e5b85b865669ce0752 (DEFAULT_TABLE table=test-table [id=ab668f18daac434cad21d08b7c27abef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:25.935994 8185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62bd76bc4b6f43e5b85b865669ce0752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:25.939163 8504 tablet_bootstrap.cc:492] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: No bootstrap required, opened a new log
I20260924 00:50:25.939427 8504 ts_tablet_manager.cc:1397] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260924 00:50:25.939785 8506 tablet_bootstrap.cc:492] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: Bootstrap starting.
I20260924 00:50:25.940392 8504 raft_consensus.cc:348] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:25.940686 8504 raft_consensus.cc:374] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:25.940814 8504 raft_consensus.cc:729] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3359189faab547c5a64c98e926d18930, State: Initialized, Role: FOLLOWER
I20260924 00:50:25.941107 8504 consensus_queue.cc:260] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:25.941599 8506 tablet_bootstrap.cc:654] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:25.943156 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.942622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:55359)
0924 00:50:25.942871 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.942967 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 00:50:25.944989 8504 ts_tablet_manager.cc:1428] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 00:50:25.946157 8506 tablet_bootstrap.cc:492] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: No bootstrap required, opened a new log
I20260924 00:50:25.946405 8506 ts_tablet_manager.cc:1397] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 00:50:25.946593 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.945796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:44803)
0924 00:50:25.946039 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.946167 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 00:50:25.946933 8109 heartbeater.cc:503] Master 127.0.222.60:35737 was elected leader, sending a full tablet report...
I20260924 00:50:25.947268 8506 raft_consensus.cc:348] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:25.947561 8506 raft_consensus.cc:374] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:25.947710 8506 raft_consensus.cc:729] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d92fed7ae44e08b7a54f52c9c03978, State: Initialized, Role: FOLLOWER
I20260924 00:50:25.949237 8506 consensus_queue.cc:260] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:25.950271 8506 ts_tablet_manager.cc:1428] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 00:50:25.951270 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.950602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:57801)
0924 00:50:25.951015 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.951121 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
I20260924 00:50:25.952435 8396 heartbeater.cc:503] Master 127.0.222.60:35737 was elected leader, sending a full tablet report...
I20260924 00:50:25.953138 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.952133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:50915)
0924 00:50:25.952838 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.952935 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578}
I20260924 00:50:25.954798 8509 tablet_bootstrap.cc:492] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: Bootstrap starting.
I20260924 00:50:25.956578 8509 tablet_bootstrap.cc:654] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:25.958747 8509 tablet_bootstrap.cc:492] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: No bootstrap required, opened a new log
I20260924 00:50:25.958910 8509 ts_tablet_manager.cc:1397] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 00:50:25.959645 8509 raft_consensus.cc:348] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:25.959861 8509 raft_consensus.cc:374] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:25.959955 8509 raft_consensus.cc:729] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ddd2e4fe40404b903b5adeda9646ae, State: Initialized, Role: FOLLOWER
I20260924 00:50:25.960181 8509 consensus_queue.cc:260] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:25.960999 8509 ts_tablet_manager.cc:1428] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:50:25.961870 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.961432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:48183)
0924 00:50:25.961591 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.961697 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 00:50:25.962827 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:25.962431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:35029)
0924 00:50:25.962595 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:25.962673 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 00:50:25.963348 8252 heartbeater.cc:503] Master 127.0.222.60:35737 was elected leader, sending a full tablet report...
I20260924 00:50:26.014462 8512 raft_consensus.cc:482] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:26.014734 8512 raft_consensus.cc:504] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:26.015620 8512 leader_election.cc:288] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755), c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573)
I20260924 00:50:26.016289 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62bd76bc4b6f43e5b85b865669ce0752" candidate_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3359189faab547c5a64c98e926d18930" is_pre_election: true
I20260924 00:50:26.016405 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62bd76bc4b6f43e5b85b865669ce0752" candidate_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" is_pre_election: true
I20260924 00:50:26.016533 8061 raft_consensus.cc:2454] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5ddd2e4fe40404b903b5adeda9646ae in term 0.
I20260924 00:50:26.016716 8349 raft_consensus.cc:2454] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5ddd2e4fe40404b903b5adeda9646ae in term 0.
I20260924 00:50:26.017110 8137 leader_election.cc:302] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3359189faab547c5a64c98e926d18930, e5ddd2e4fe40404b903b5adeda9646ae; no voters:
I20260924 00:50:26.017436 8512 raft_consensus.cc:2781] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:26.017555 8512 raft_consensus.cc:482] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:26.017650 8512 raft_consensus.cc:3037] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:26.019912 8512 raft_consensus.cc:504] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:26.020541 8512 leader_election.cc:288] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [CANDIDATE]: Term 1 election: Requested vote from peers 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755), c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573)
I20260924 00:50:26.021282 8349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62bd76bc4b6f43e5b85b865669ce0752" candidate_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d92fed7ae44e08b7a54f52c9c03978"
I20260924 00:50:26.021317 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62bd76bc4b6f43e5b85b865669ce0752" candidate_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3359189faab547c5a64c98e926d18930"
I20260924 00:50:26.021505 8349 raft_consensus.cc:3037] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:26.021564 8061 raft_consensus.cc:3037] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:26.023849 8349 raft_consensus.cc:2454] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5ddd2e4fe40404b903b5adeda9646ae in term 1.
I20260924 00:50:26.024333 8139 leader_election.cc:302] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1d92fed7ae44e08b7a54f52c9c03978, e5ddd2e4fe40404b903b5adeda9646ae; no voters:
I20260924 00:50:26.024649 8512 raft_consensus.cc:2781] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:26.024673 8061 raft_consensus.cc:2454] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5ddd2e4fe40404b903b5adeda9646ae in term 1.
I20260924 00:50:26.025883 8512 raft_consensus.cc:686] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 LEADER]: Becoming Leader. State: Replica: e5ddd2e4fe40404b903b5adeda9646ae, State: Running, Role: LEADER
I20260924 00:50:26.026520 8512 consensus_queue.cc:237] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } }
I20260924 00:50:26.027155 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.026684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:53795)
0924 00:50:26.026885 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:26.026984 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 00:50:26.029364 7922 catalog_manager.cc:5495] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae reported cstate change: term changed from 0 to 1, leader changed from <none> to e5ddd2e4fe40404b903b5adeda9646ae (127.0.222.2). New cstate: current_term: 1 leader_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5ddd2e4fe40404b903b5adeda9646ae" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40137 } health_report { overall_health: HEALTHY } } }
I20260924 00:50:26.062548 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.057915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.0.222.2:40137)
0924 00:50:26.058142 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:26.058148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:26.058168 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:50:26.058231 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:26.058234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:26.058329 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:26.058401 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:26.058703 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:26.059426 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:26.060815 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:26.061302 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:26.061407 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:26.061826 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:26.061940 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:26.062018 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:26.062195 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 00:50:26.062273 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 00:50:26.066068 8061 raft_consensus.cc:1261] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 1 FOLLOWER]: Refusing update from remote peer e5ddd2e4fe40404b903b5adeda9646ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:26.066777 8349 raft_consensus.cc:1261] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 1 FOLLOWER]: Refusing update from remote peer e5ddd2e4fe40404b903b5adeda9646ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:26.067803 8513 consensus_queue.cc:1035] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d92fed7ae44e08b7a54f52c9c03978" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:26.068485 8513 consensus_queue.cc:1035] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "3359189faab547c5a64c98e926d18930" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:26.073128 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.072485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:59389)
0924 00:50:26.072753 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:26.072940 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260924 00:50:26.073688 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.073004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:59617)
0924 00:50:26.073439 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:26.073536 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
I20260924 00:50:26.078975 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.077788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.1:40355)
0924 00:50:26.078646 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:26.078755 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725}
I20260924 00:50:26.082429 8517 mvcc.cc:205] Tried to move back new op lower bound from 7332704362759352320 to 7332704362607185920. Current Snapshot: MvccSnapshot[applied={T|T < 7332704362759352320}]
I20260924 00:50:26.084489 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.083545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.3:60027)
0924 00:50:26.084158 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:26.084266 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469}
I20260924 00:50:26.086175 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.085649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.222.2:41553)
0924 00:50:26.085848 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:26.085958 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 00:50:26.089151 7923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:26.089715 7923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:26.095614 7923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3986ed3b261f467fa0d79609b31b073c: Sending DeleteTablet for 3 replicas of tablet 62bd76bc4b6f43e5b85b865669ce0752
I20260924 00:50:26.096674 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 62bd76bc4b6f43e5b85b865669ce0752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:26 UTC) from {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:37360
I20260924 00:50:26.096691 8329 tablet_service.cc:1514] Processing DeleteTablet for tablet 62bd76bc4b6f43e5b85b865669ce0752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:26 UTC) from {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:47772
I20260924 00:50:26.096935 8185 tablet_service.cc:1514] Processing DeleteTablet for tablet 62bd76bc4b6f43e5b85b865669ce0752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:26 UTC) from {username='oryx', principal='oryx/127.0.222.60@KRBTEST.COM'} at 127.0.0.1:37676
I20260924 00:50:26.097455 8524 tablet_replica.cc:330] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: stopping tablet replica
I20260924 00:50:26.097657 8524 raft_consensus.cc:2229] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:26.097752 8526 tablet_replica.cc:330] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: stopping tablet replica
I20260924 00:50:26.097914 8526 raft_consensus.cc:2229] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:50:26.097992 8524 raft_consensus.cc:2258] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:26.098407 8526 raft_consensus.cc:2258] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:26.098493 8525 tablet_replica.cc:330] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: stopping tablet replica
I20260924 00:50:26.098897 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 7894
I20260924 00:50:26.099061 8525 raft_consensus.cc:2229] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:26.099324 8525 raft_consensus.cc:2258] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:26.100209 8526 ts_tablet_manager.cc:1905] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:26.100840 8525 ts_tablet_manager.cc:1905] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:26.102638 8524 ts_tablet_manager.cc:1905] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:26.107432 8525 ts_tablet_manager.cc:1918] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:26.107879 8526 ts_tablet_manager.cc:1918] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:26.108079 8526 log.cc:1196] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/62bd76bc4b6f43e5b85b865669ce0752
I20260924 00:50:26.108418 8525 log.cc:1196] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/62bd76bc4b6f43e5b85b865669ce0752
I20260924 00:50:26.108544 8526 ts_tablet_manager.cc:1939] T 62bd76bc4b6f43e5b85b865669ce0752 P e5ddd2e4fe40404b903b5adeda9646ae: Deleting consensus metadata
I20260924 00:50:26.108979 8525 ts_tablet_manager.cc:1939] T 62bd76bc4b6f43e5b85b865669ce0752 P c1d92fed7ae44e08b7a54f52c9c03978: Deleting consensus metadata
I20260924 00:50:26.109490 8524 ts_tablet_manager.cc:1918] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:26.109668 8524 log.cc:1196] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/62bd76bc4b6f43e5b85b865669ce0752
I20260924 00:50:26.109918 7910 catalog_manager.cc:4841] TS e5ddd2e4fe40404b903b5adeda9646ae (127.0.222.2:40137): tablet 62bd76bc4b6f43e5b85b865669ce0752 (table test-table [id=ab668f18daac434cad21d08b7c27abef]) successfully deleted
I20260924 00:50:26.110117 8524 ts_tablet_manager.cc:1939] T 62bd76bc4b6f43e5b85b865669ce0752 P 3359189faab547c5a64c98e926d18930: Deleting consensus metadata
I20260924 00:50:26.110221 7909 catalog_manager.cc:4841] TS c1d92fed7ae44e08b7a54f52c9c03978 (127.0.222.3:33573): tablet 62bd76bc4b6f43e5b85b865669ce0752 (table test-table [id=ab668f18daac434cad21d08b7c27abef]) successfully deleted
I20260924 00:50:26.111070 7907 catalog_manager.cc:4841] TS 3359189faab547c5a64c98e926d18930 (127.0.222.1:46755): tablet 62bd76bc4b6f43e5b85b865669ce0752 (table test-table [id=ab668f18daac434cad21d08b7c27abef]) successfully deleted
I20260924 00:50:26.122130 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.100165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.0.222.1:46755)
0924 00:50:26.100359 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:26.100365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:26.108166 (+ 7801us) server_negotiation.cc:407] Connection header received
0924 00:50:26.108388 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:26.108394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:26.108491 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:26.108598 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:26.110506 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:26.111390 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:26.112579 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:26.112996 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:26.113105 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:26.115939 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:26.115957 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:26.115974 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:26.116010 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:26.119019 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:26.119828 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:26.119837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:26.119839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:26.120004 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:26.120434 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:26.120438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:26.120440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:26.120837 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:26.121130 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:26.121549 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 00:50:26.121804 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:50:26.122941 8103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:26.223775 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 7894
I20260924 00:50:26.238317 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 8121
I20260924 00:50:26.251262 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.239155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.0.222.2:40137)
0924 00:50:26.239342 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:26.239348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:26.239367 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:50:26.239497 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:26.239501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:26.239572 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:26.239653 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:26.241055 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:26.241909 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:26.243289 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:26.243682 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:26.243776 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:26.246200 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:26.246215 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:26.246231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:26.246267 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:26.248815 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:26.249516 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:26.249523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:26.249525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:26.249661 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:26.249957 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:26.249960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:26.249962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:26.250315 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:26.250525 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:26.250779 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 00:50:26.250954 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:50:26.252041 8246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:26.353049 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.352521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:47044)
0924 00:50:26.352706 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:26.352850 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 00:50:26.354184 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.353762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42943 (local address 127.0.0.1:47046)
0924 00:50:26.353955 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:26.354040 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 00:50:26.372695 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 8121
I20260924 00:50:26.391436 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 8264
I20260924 00:50:26.404865 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:26.392336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.0.222.3:33573)
0924 00:50:26.392527 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:26.392531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:26.392547 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:50:26.392602 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:26.392606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:26.392673 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:26.392775 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:26.394211 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:26.394919 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:26.396093 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:26.396532 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:26.396631 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:26.399323 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:26.399339 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:26.399355 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:26.399388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:26.402284 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:26.402944 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:26.402952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:26.402954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:26.403091 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:26.403480 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:26.403483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:26.403485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:26.403837 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:26.404094 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:26.404391 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 00:50:26.404558 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:50:26.405680 8390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:26.499159 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 8264
I20260924 00:50:26.513353 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 7585
I20260924 00:50:26.514554 7761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:26.636368 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 7585
I20260924 00:50:26.652997 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 7682
I20260924 00:50:26.654181 7953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:26.783622 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 7682
2026-09-24T00:50:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12053 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:50:26 dist-test-slave-rtvz krb5kdc[8536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:50:26 dist-test-slave-rtvz krb5kdc[8536](info): set up 2 sockets
Sep 24 00:50:26 dist-test-slave-rtvz krb5kdc[8536](info): commencing operation
krb5kdc: starting...
W20260924 00:50:27.817375 888 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.005s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:50:27 dist-test-slave-rtvz krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.62.keytab.
Entry for principal oryx/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.62.keytab.
I20260924 00:50:28.032582 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41957
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.62.keytab
--principal=oryx/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:28.362522 8549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:28.362864 8549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:28.363009 8549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:28.370357 8549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:28.370489 8549 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:50:28.370549 8549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:28.370640 8549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:28.370707 8549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:28.370800 8549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:28.380985 8549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41957
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.62.keytab
--principal=oryx/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:28.382364 8549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:28.384037 8549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:28.396838 8555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:28.396996 8549 server_base.cc:1034] running on GCE node
W20260924 00:50:28.396831 8557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:28.396836 8554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:28.398219 8549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:28.399626 8549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:28.400862 8549 hybrid_clock.cc:648] HybridClock initialized: now 1790211028400796 us; error 43 us; skew 500 ppm
I20260924 00:50:28.401214 8549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:28 dist-test-slave-rtvz krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211028, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:28.409646 8549 init.cc:377] Logged in from keytab as oryx/127.0.222.62@KRBTEST.COM (short username oryx)
I20260924 00:50:28.411337 8549 webserver.cc:460] Webserver started at http://127.0.222.62:34853/ using document root <none> and password file <none>
I20260924 00:50:28.412092 8549 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:28.412202 8549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:28.412540 8549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:28.415696 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "e936dc660af14ad882b69ef2fb28b805"
format_stamp: "Formatted at 2026-09-24 00:50:28 on dist-test-slave-rtvz"
server_key: "41031cf3f3e4a4cbe430d19c7e1110c8"
server_key_iv: "70a1dc13541a93832865b00c7bed475b"
server_key_version: "encryptionkey@0"
I20260924 00:50:28.416509 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "e936dc660af14ad882b69ef2fb28b805"
format_stamp: "Formatted at 2026-09-24 00:50:28 on dist-test-slave-rtvz"
server_key: "41031cf3f3e4a4cbe430d19c7e1110c8"
server_key_iv: "70a1dc13541a93832865b00c7bed475b"
server_key_version: "encryptionkey@0"
I20260924 00:50:28.421720 8549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 00:50:28.425370 8564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:28.426862 8549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:50:28.427030 8549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "e936dc660af14ad882b69ef2fb28b805"
format_stamp: "Formatted at 2026-09-24 00:50:28 on dist-test-slave-rtvz"
server_key: "41031cf3f3e4a4cbe430d19c7e1110c8"
server_key_iv: "70a1dc13541a93832865b00c7bed475b"
server_key_version: "encryptionkey@0"
I20260924 00:50:28.427235 8549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:28.455681 8549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:28.460078 8549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:28.460376 8549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:28.481516 8549 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41957
I20260924 00:50:28.481535 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41957 every 8 connection(s)
I20260924 00:50:28.482864 8549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:28.483368 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 8549
I20260924 00:50:28.483601 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:28.484481 888 external_mini_cluster.cc:1428] Setting key 6b2936d9d9ce8ee1ce1afbb6543b3ae2
I20260924 00:50:28.487242 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 00:50:28 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:28.500036 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: Bootstrap starting.
I20260924 00:50:28.504172 8627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:28.505513 8627 log.cc:824] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:28.509002 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: No bootstrap required, opened a new log
I20260924 00:50:28.509241 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:28.486555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.0.222.62:41957)
0924 00:50:28.487403 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:28.487420 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:28.487465 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:50:28.489332 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:28.489355 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:28.489776 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:28.490162 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:28.491867 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:28.493297 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:28.494996 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:28.495731 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:28.495886 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:28.500556 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:28.500566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:28.500587 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:28.500637 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:28.504453 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:28.505249 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:28.505258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:28.505268 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:28.505490 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:28.505981 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:28.505986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:28.505989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:28.506661 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:28.506902 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:28.507432 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 00:50:28.507879 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":367,"threads_started":1}
I20260924 00:50:28.517663 8627 raft_consensus.cc:348] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e936dc660af14ad882b69ef2fb28b805" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41957 } }
I20260924 00:50:28.518076 8627 raft_consensus.cc:374] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:28.518158 8627 raft_consensus.cc:729] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e936dc660af14ad882b69ef2fb28b805, State: Initialized, Role: FOLLOWER
I20260924 00:50:28.518738 8627 consensus_queue.cc:260] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e936dc660af14ad882b69ef2fb28b805" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41957 } }
I20260924 00:50:28.518947 8627 raft_consensus.cc:388] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:28.519050 8627 raft_consensus.cc:482] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:28.519174 8627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:28.521492 8627 raft_consensus.cc:504] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e936dc660af14ad882b69ef2fb28b805" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41957 } }
I20260924 00:50:28.522080 8627 leader_election.cc:302] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e936dc660af14ad882b69ef2fb28b805; no voters:
I20260924 00:50:28.522873 8627 leader_election.cc:288] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:28.523000 8632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:28.523783 8632 raft_consensus.cc:686] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [term 1 LEADER]: Becoming Leader. State: Replica: e936dc660af14ad882b69ef2fb28b805, State: Running, Role: LEADER
I20260924 00:50:28.524358 8632 consensus_queue.cc:237] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e936dc660af14ad882b69ef2fb28b805" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41957 } }
I20260924 00:50:28.524875 8627 sys_catalog.cc:564] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:28.527130 8634 sys_catalog.cc:455] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e936dc660af14ad882b69ef2fb28b805. Latest consensus state: current_term: 1 leader_uuid: "e936dc660af14ad882b69ef2fb28b805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e936dc660af14ad882b69ef2fb28b805" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41957 } } }
I20260924 00:50:28.527513 8634 sys_catalog.cc:458] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:28.527956 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e936dc660af14ad882b69ef2fb28b805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e936dc660af14ad882b69ef2fb28b805" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41957 } } }
I20260924 00:50:28.528218 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805 [sys.catalog]: This master's current role is: LEADER
W20260924 00:50:28.540238 8648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:50:28.540382 8648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:50:28.540555 8637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:28.542810 8637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:28.552289 8637 catalog_manager.cc:1347] Generated new cluster ID: 22116b0e80c14968a7e787144162daf7
I20260924 00:50:28.552423 8637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:28.568185 8637 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:28.570160 8637 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:28.583505 8637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: Generated new TSK 0
I20260924 00:50:28.584638 8637 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab.
I20260924 00:50:28.645407 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41957
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:28.983605 8653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:28.983933 8653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:28.984058 8653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:28.992355 8653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:28.992506 8653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:28.992642 8653 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:50:28.992760 8653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:29.003669 8653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:29.005017 8653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:29.006466 8653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:29.021190 8658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:29.021317 8659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:29.021245 8661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:29.022086 8653 server_base.cc:1034] running on GCE node
I20260924 00:50:29.022693 8653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:29.023638 8653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:29.024926 8653 hybrid_clock.cc:648] HybridClock initialized: now 1790211029024778 us; error 49 us; skew 500 ppm
I20260924 00:50:29.025277 8653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:29 dist-test-slave-rtvz krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:29.028411 8653 init.cc:377] Logged in from keytab as oryx/127.0.222.1@KRBTEST.COM (short username oryx)
I20260924 00:50:29.030100 8653 webserver.cc:460] Webserver started at http://127.0.222.1:34221/ using document root <none> and password file <none>
I20260924 00:50:29.030824 8653 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:29.030920 8653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:29.031193 8653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:29.034235 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "90f2d969822848c98c8fb04a91d1c2b0"
format_stamp: "Formatted at 2026-09-24 00:50:29 on dist-test-slave-rtvz"
server_key: "94a38f42fc8eb45910cfa2752a87b0e0"
server_key_iv: "2996b5b04015f88a32234a41cc14032b"
server_key_version: "encryptionkey@0"
I20260924 00:50:29.035041 8653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "90f2d969822848c98c8fb04a91d1c2b0"
format_stamp: "Formatted at 2026-09-24 00:50:29 on dist-test-slave-rtvz"
server_key: "94a38f42fc8eb45910cfa2752a87b0e0"
server_key_iv: "2996b5b04015f88a32234a41cc14032b"
server_key_version: "encryptionkey@0"
I20260924 00:50:29.040690 8653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:50:29.044746 8668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:29.046388 8653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:29.046562 8653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "90f2d969822848c98c8fb04a91d1c2b0"
format_stamp: "Formatted at 2026-09-24 00:50:29 on dist-test-slave-rtvz"
server_key: "94a38f42fc8eb45910cfa2752a87b0e0"
server_key_iv: "2996b5b04015f88a32234a41cc14032b"
server_key_version: "encryptionkey@0"
I20260924 00:50:29.046816 8653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:29.071214 8653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:29.076333 8653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:29.076624 8653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:29.079660 8653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:29.079779 8653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:29.079938 8653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:29.080003 8653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:50:29 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:29.104931 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:29.080517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45155 (local address 127.0.222.62:41957)
0924 00:50:29.081085 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:29.081089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:29.082409 (+ 1320us) server_negotiation.cc:407] Connection header received
0924 00:50:29.084580 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:29.084585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:29.084674 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:29.084776 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:29.087226 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.088202 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.090860 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.091365 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.091502 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:29.097142 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:29.097150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:29.097155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:29.097229 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:29.100486 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:29.101603 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:29.101609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:29.101613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:29.101804 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:29.102494 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:29.102502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:29.102506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:29.102973 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:29.103165 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:29.104284 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0924 00:50:29.104554 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":357,"threads_started":1}
I20260924 00:50:29.106187 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:29.081299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41957 (local address 127.0.222.1:45155)
0924 00:50:29.082238 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:29.082278 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:29.084279 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:29.085128 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:29.085142 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:29.085995 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:29.086949 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:29.086992 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.088383 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.088388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:29.090687 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:29.090705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.091674 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.091682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:29.092003 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:29.092911 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:29.092948 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:29.096750 (+ 3802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:29.100674 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:29.100687 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:29.100711 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:29.101432 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:29.101969 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:29.101975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:29.101977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:29.102311 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:29.103152 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:29.103159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:29.103664 (+ 505us) client_negotiation.cc:769] Sending connection context
0924 00:50:29.104089 (+ 425us) client_negotiation.cc:240] Negotiation successful
0924 00:50:29.104817 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":409,"threads_started":1}
I20260924 00:50:29.128101 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:29.112801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41957 (local address 127.0.222.1:47097)
0924 00:50:29.113548 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:29.113569 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:29.113755 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:29.114079 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:29.114084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:29.114750 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:29.115198 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:29.115214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.116327 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.116332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:29.118232 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:29.118261 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.119327 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.119341 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:29.119526 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:29.120291 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:29.120308 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:29.121141 (+ 833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:29.124988 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:29.124998 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:29.125002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:29.125715 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:29.126278 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:29.126282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:29.126286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:29.126452 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:29.127099 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:29.127102 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:29.127302 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 00:50:29.127429 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 00:50:29.127731 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":355,"threads_started":1}
I20260924 00:50:29.128259 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:29.112661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:47097 (local address 127.0.222.62:41957)
0924 00:50:29.112858 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:29.112865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:29.113679 (+ 814us) server_negotiation.cc:407] Connection header received
0924 00:50:29.113879 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:29.113884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:29.113950 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:29.114031 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:29.115417 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.116165 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.118434 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.118939 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.119078 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:29.121398 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:29.121405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:29.121410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:29.121457 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:29.124820 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:29.125936 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:29.125942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:29.125947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:29.126146 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:29.126565 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:29.126570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:29.126574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:29.126986 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:29.127153 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:29.127574 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 00:50:29.127851 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:50:29.132120 8653 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:33361
I20260924 00:50:29.132139 8789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:33361 every 8 connection(s)
I20260924 00:50:29.133567 8653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:29.135669 8790 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41957
I20260924 00:50:29.135921 8790 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:29.136601 8790 heartbeater.cc:511] Master 127.0.222.62:41957 requested a full tablet report, sending...
I20260924 00:50:29.138646 8581 ts_manager.cc:194] Registered new tserver with Master: 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361)
I20260924 00:50:29.139902 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 8653
I20260924 00:50:29.140086 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:50:29.140161 8581 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:45155
I20260924 00:50:29.140704 888 external_mini_cluster.cc:1428] Setting key be89a568d6a49e733ae5885f00ad9aca
WARNING: no policy specified for oryx/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab.
I20260924 00:50:29.188514 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41957
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:29.515666 8794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:29.516003 8794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:29.516155 8794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:29.523794 8794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:29.523943 8794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:29.524065 8794 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:50:29.524174 8794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:29.534555 8794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:29.535907 8794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:29.537328 8794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:29.550913 8799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:29.550829 8802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:29.550819 8800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:29.551851 8794 server_base.cc:1034] running on GCE node
I20260924 00:50:29.552526 8794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:29.553544 8794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:29.554766 8794 hybrid_clock.cc:648] HybridClock initialized: now 1790211029554690 us; error 60 us; skew 500 ppm
I20260924 00:50:29.555140 8794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:29 dist-test-slave-rtvz krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:29.558287 8794 init.cc:377] Logged in from keytab as oryx/127.0.222.2@KRBTEST.COM (short username oryx)
I20260924 00:50:29.559739 8794 webserver.cc:460] Webserver started at http://127.0.222.2:34835/ using document root <none> and password file <none>
I20260924 00:50:29.560426 8794 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:29.560513 8794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:29.560794 8794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:29.563793 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
format_stamp: "Formatted at 2026-09-24 00:50:29 on dist-test-slave-rtvz"
server_key: "1814c281f4a8681af535cb0346e65777"
server_key_iv: "0984767b632fa1b868bc468c7ccd0a9d"
server_key_version: "encryptionkey@0"
I20260924 00:50:29.564550 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
format_stamp: "Formatted at 2026-09-24 00:50:29 on dist-test-slave-rtvz"
server_key: "1814c281f4a8681af535cb0346e65777"
server_key_iv: "0984767b632fa1b868bc468c7ccd0a9d"
server_key_version: "encryptionkey@0"
I20260924 00:50:29.569957 8794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:50:29.573835 8809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:29.575371 8794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 00:50:29.575536 8794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
format_stamp: "Formatted at 2026-09-24 00:50:29 on dist-test-slave-rtvz"
server_key: "1814c281f4a8681af535cb0346e65777"
server_key_iv: "0984767b632fa1b868bc468c7ccd0a9d"
server_key_version: "encryptionkey@0"
I20260924 00:50:29.575762 8794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:29.601724 8794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:29.606928 8794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:29.607205 8794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:29.610405 8794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:29.610517 8794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:29.610690 8794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:29.610822 8794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 00:50:29 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211029, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:29.635767 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:29.611005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:60639 (local address 127.0.222.62:41957)
0924 00:50:29.611267 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:29.611276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:29.612788 (+ 1512us) server_negotiation.cc:407] Connection header received
0924 00:50:29.614994 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:29.615002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:29.615123 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:29.615267 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:29.617750 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.618941 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.621497 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.622250 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.622430 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:29.627402 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:29.627410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:29.627414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:29.627464 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:29.631301 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:29.632456 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:29.632462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:29.632466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:29.632606 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:29.633295 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:29.633301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:29.633304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:29.633769 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:29.633949 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:29.635298 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0924 00:50:29.635503 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 00:50:29.637950 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:29.611697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41957 (local address 127.0.222.2:60639)
0924 00:50:29.612576 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:29.612633 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:29.614673 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:29.615605 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:29.615616 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:29.616451 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:29.617459 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:29.617496 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.619165 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.619171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:29.621296 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:29.621317 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.622553 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.622562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:29.622864 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:29.623642 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:29.623668 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:29.627081 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:29.631492 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:29.631500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:29.631516 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:29.632304 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:29.632734 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:29.632739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:29.632743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:29.633079 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:29.633961 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:29.633977 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:29.634609 (+ 632us) client_negotiation.cc:769] Sending connection context
0924 00:50:29.635189 (+ 580us) client_negotiation.cc:240] Negotiation successful
0924 00:50:29.635909 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":371,"threads_started":1}
I20260924 00:50:29.657442 8794 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:33283
I20260924 00:50:29.657450 8929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:33283 every 8 connection(s)
I20260924 00:50:29.659123 8794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:29.659493 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 8794
I20260924 00:50:29.659669 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:50:29.660194 888 external_mini_cluster.cc:1428] Setting key 323ee8abde824230df1fe1296ccc7d5d
I20260924 00:50:29.660794 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:29.645835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41957 (local address 127.0.222.2:50091)
0924 00:50:29.646468 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:29.646484 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:29.646639 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:29.646987 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:29.646991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:29.647490 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:29.647786 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:29.647795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.648834 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.648839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:29.650190 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:29.650215 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.651455 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.651463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:29.651685 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:29.652391 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:29.652420 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:29.653552 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:29.657803 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:29.657807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:29.657810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:29.658226 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:29.658665 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:29.658668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:29.658670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:29.658807 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:29.659604 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:29.659609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:29.659825 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 00:50:29.659978 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 00:50:29.660347 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":366,"threads_started":1}
I20260924 00:50:29.660818 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:29.645691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50091 (local address 127.0.222.62:41957)
0924 00:50:29.645911 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:29.645915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:29.646587 (+ 672us) server_negotiation.cc:407] Connection header received
0924 00:50:29.646770 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:29.646776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:29.646857 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:29.646947 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:29.647939 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.648688 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.650443 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:29.651217 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:29.651379 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:29.653798 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:29.653807 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:29.653814 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:29.653866 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:29.657643 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:29.658362 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:29.658368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:29.658372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:29.658553 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:29.658915 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:29.658920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:29.658924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:29.659464 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:29.659636 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:29.660125 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 00:50:29.660403 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 00:50:29.665972 8930 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41957
I20260924 00:50:29.666210 8930 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:29.666973 8930 heartbeater.cc:511] Master 127.0.222.62:41957 requested a full tablet report, sending...
I20260924 00:50:29.668497 8580 ts_manager.cc:194] Registered new tserver with Master: 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283)
I20260924 00:50:29.669508 8580 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:60639
WARNING: no policy specified for oryx/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab.
I20260924 00:50:29.710065 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:41957
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:30.043825 8934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:30.044147 8934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:30.044289 8934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:30.052062 8934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:30.052206 8934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:30.052340 8934 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:50:30.052454 8934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:30.068203 8934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.222.62:41957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:30.070111 8934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:30.072026 8934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:30.089857 8939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:30.090058 8942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:30.089819 8940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:30.093354 8934 server_base.cc:1034] running on GCE node
I20260924 00:50:30.094396 8934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:30.095875 8934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:30.097246 8934 hybrid_clock.cc:648] HybridClock initialized: now 1790211030097115 us; error 63 us; skew 500 ppm
I20260924 00:50:30.097671 8934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:30 dist-test-slave-rtvz krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:30.102151 8934 init.cc:377] Logged in from keytab as oryx/127.0.222.3@KRBTEST.COM (short username oryx)
I20260924 00:50:30.104146 8934 webserver.cc:460] Webserver started at http://127.0.222.3:44955/ using document root <none> and password file <none>
I20260924 00:50:30.105189 8934 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:30.105366 8934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:30.105800 8934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:30.110395 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "1bec43e620c64ca0aba75da41c21b190"
format_stamp: "Formatted at 2026-09-24 00:50:30 on dist-test-slave-rtvz"
server_key: "dda32439b0df691382f8133ad2db35f3"
server_key_iv: "4f63fd886b9d01a345d2808bd34fb0ad"
server_key_version: "encryptionkey@0"
I20260924 00:50:30.111361 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "1bec43e620c64ca0aba75da41c21b190"
format_stamp: "Formatted at 2026-09-24 00:50:30 on dist-test-slave-rtvz"
server_key: "dda32439b0df691382f8133ad2db35f3"
server_key_iv: "4f63fd886b9d01a345d2808bd34fb0ad"
server_key_version: "encryptionkey@0"
I20260924 00:50:30.116788 8934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 00:50:30.120786 8949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:30.122351 8934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:30.122534 8934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "1bec43e620c64ca0aba75da41c21b190"
format_stamp: "Formatted at 2026-09-24 00:50:30 on dist-test-slave-rtvz"
server_key: "dda32439b0df691382f8133ad2db35f3"
server_key_iv: "4f63fd886b9d01a345d2808bd34fb0ad"
server_key_version: "encryptionkey@0"
I20260924 00:50:30.122828 8934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:30.143638 8790 heartbeater.cc:503] Master 127.0.222.62:41957 was elected leader, sending a full tablet report...
I20260924 00:50:30.145871 8934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:30.150326 8934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:30.150591 8934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:30.153616 8934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:30.153723 8934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:30.153870 8934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:30.153935 8934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:50:30 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211030, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:30.178043 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.154243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:33795 (local address 127.0.222.62:41957)
0924 00:50:30.154440 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.154447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.156197 (+ 1750us) server_negotiation.cc:407] Connection header received
0924 00:50:30.158414 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.158420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.158509 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.158599 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:30.161056 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.162023 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.164763 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.165462 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.165595 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.170187 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:30.170196 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:30.170202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:30.170256 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:30.174196 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:30.175295 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:30.175302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:30.175306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:30.175456 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:30.176073 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:30.176079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:30.176082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:30.176483 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:30.176627 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.177419 (+ 792us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.177674 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:50:30.179160 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.155036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41957 (local address 127.0.222.3:33795)
0924 00:50:30.155994 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.156045 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.158091 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.158905 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.158918 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.159798 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:30.160826 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.160862 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.162190 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.162196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.164575 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.164591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.165750 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.165759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.165998 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.166862 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:30.166891 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:30.169781 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:30.174430 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:30.174441 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:30.174462 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:30.175137 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:30.175584 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:30.175589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:30.175593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:30.175950 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:30.176628 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:30.176637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:30.177009 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.177312 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.177897 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":400,"threads_started":1}
I20260924 00:50:30.201061 8934 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:37627
I20260924 00:50:30.201081 9071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:37627 every 8 connection(s)
I20260924 00:50:30.202610 8934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:30.203022 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.187988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:46981 (local address 127.0.222.62:41957)
0924 00:50:30.188206 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.188213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.188429 (+ 216us) server_negotiation.cc:407] Connection header received
0924 00:50:30.188627 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.188633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.188724 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.188824 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:30.190544 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.191618 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.193260 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.193720 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.193839 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.196448 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:30.196455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:30.196461 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:30.196507 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:30.200052 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:30.201040 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:30.201045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:30.201048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:30.201192 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:30.201622 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:30.201626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:30.201628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:30.201940 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:30.202052 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.202531 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.202737 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 00:50:30.203259 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.187416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41957 (local address 127.0.222.3:46981)
0924 00:50:30.188309 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.188328 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.188493 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.189041 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.189047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.189823 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:30.190313 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.190329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.191833 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.191838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.193088 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.193104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.193911 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.193917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.194104 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.194885 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:30.194909 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:30.196215 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:30.200226 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:30.200232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:30.200237 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:30.200852 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:30.201348 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:30.201353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:30.201357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:30.201501 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:30.202073 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:30.202077 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:30.202299 (+ 222us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.202445 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.202785 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":786,"threads_started":1}
I20260924 00:50:30.203296 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 8934
I20260924 00:50:30.203486 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:50:30.203958 888 external_mini_cluster.cc:1428] Setting key f7890e139af54339a8d23910f8f11fd9
I20260924 00:50:30.207866 9072 heartbeater.cc:347] Connected to a master server at 127.0.222.62:41957
I20260924 00:50:30.208107 9072 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:30.208755 9072 heartbeater.cc:511] Master 127.0.222.62:41957 requested a full tablet report, sending...
I20260924 00:50:30.210134 8580 ts_manager.cc:194] Registered new tserver with Master: 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627)
I20260924 00:50:30.211015 8580 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:33795
I20260924 00:50:30.219578 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:50:30 dist-test-slave-rtvz krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:50:30 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:30 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:50:30.267378 8677 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:30.256676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.0.222.62:41957)
0924 00:50:30.256909 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.256914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.257139 (+ 225us) server_negotiation.cc:407] Connection header received
0924 00:50:30.257346 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.257351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.257433 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.257531 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:30.258949 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.259694 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.261038 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.261548 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.261721 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.267065 (+ 5344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37490: BlockingRecv error: recv got EOF from 127.0.0.1:37490 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 24 00:50:30 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211030, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:30.283324 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.270320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.0.222.62:41957)
0924 00:50:30.270533 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.270538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.270724 (+ 186us) server_negotiation.cc:407] Connection header received
0924 00:50:30.270882 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.270886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.270960 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.271040 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:30.272141 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.272876 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.274144 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.274561 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.274660 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.278111 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:30.278116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:30.278119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:30.278153 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:30.280742 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:30.281425 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:30.281429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:30.281431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:30.281547 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:30.281905 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:30.281908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:30.281910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:30.282252 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:30.282531 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.282828 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.283026 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:50:30.287353 8580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:50:30.294636 8580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:30.318214 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.307388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.0.222.3:37627)
0924 00:50:30.308841 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.308849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.308882 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:50:30.308984 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.308992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.309373 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.310289 (+ 916us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:30.310620 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.312324 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.315106 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.316938 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.317064 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.317356 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.317707 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.317886 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":1101,"threads_started":1}
I20260924 00:50:30.318305 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.307089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:37627 (local address 127.0.0.1:50188)
0924 00:50:30.307883 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.307932 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.308100 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.309644 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.309652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.309686 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:30.310065 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.310079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.312486 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.312492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.314957 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.314971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.317155 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.317190 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.317476 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.317520 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.317646 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.317854 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":51,"thread_start_us":376,"threads_started":1}
I20260924 00:50:30.320627 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.307534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33361 (local address 127.0.0.1:49152)
0924 00:50:30.309115 (+ 1581us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.309132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.309274 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.313401 (+ 4127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.313405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.313437 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:30.313943 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.313955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.315726 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.315731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.318076 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.318095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.319765 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.319772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.319955 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.319990 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.320066 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.320206 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1414,"mutex_wait_us":51,"thread_start_us":5772,"threads_started":1}
I20260924 00:50:30.321309 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.308399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33283 (local address 127.0.0.1:55334)
0924 00:50:30.313620 (+ 5221us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.313645 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.313769 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.314349 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.314353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.314377 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:30.314869 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.314881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.316368 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.316373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.318595 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.318608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.320491 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.320498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.320671 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.320702 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.320789 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.320916 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5096,"mutex_wait_us":41,"thread_start_us":640,"threads_started":1}
I20260924 00:50:30.321792 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.308089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.0.222.1:33361)
0924 00:50:30.312566 (+ 4477us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.312574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.312604 (+ 30us) server_negotiation.cc:407] Connection header received
0924 00:50:30.312703 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.312711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.313057 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.313951 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:30.314142 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.315580 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.318300 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.319540 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.321035 (+ 1495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.321282 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.321358 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.321520 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4320,"thread_start_us":4200,"threads_started":1}
I20260924 00:50:30.322009 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.308894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.0.222.2:33283)
0924 00:50:30.310039 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.310046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.313759 (+ 3713us) server_negotiation.cc:407] Connection header received
0924 00:50:30.313910 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.313918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.314159 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.314344 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:30.315054 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.316197 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.318773 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.319989 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.320117 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.320326 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.321538 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.321712 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":896,"threads_started":1}
I20260924 00:50:30.323079 9001 tablet_service.cc:1467] Processing CreateTablet for tablet 5075f1d24b634ad38e63f2c38474790d (DEFAULT_TABLE table=test-table [id=6f5039aa904449c188c19b8081b261ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:30.324767 9001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075f1d24b634ad38e63f2c38474790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:30.325053 8860 tablet_service.cc:1467] Processing CreateTablet for tablet 5075f1d24b634ad38e63f2c38474790d (DEFAULT_TABLE table=test-table [id=6f5039aa904449c188c19b8081b261ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:30.325376 8720 tablet_service.cc:1467] Processing CreateTablet for tablet 5075f1d24b634ad38e63f2c38474790d (DEFAULT_TABLE table=test-table [id=6f5039aa904449c188c19b8081b261ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:30.326473 8860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075f1d24b634ad38e63f2c38474790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:30.326707 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075f1d24b634ad38e63f2c38474790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:30.338114 9097 tablet_bootstrap.cc:492] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Bootstrap starting.
I20260924 00:50:30.341995 9099 tablet_bootstrap.cc:492] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Bootstrap starting.
I20260924 00:50:30.342973 9097 tablet_bootstrap.cc:654] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:30.343510 9098 tablet_bootstrap.cc:492] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Bootstrap starting.
I20260924 00:50:30.344491 9097 log.cc:824] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:30.345539 9099 tablet_bootstrap.cc:654] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:30.346616 9099 log.cc:824] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:30.346820 9098 tablet_bootstrap.cc:654] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:30.347754 9097 tablet_bootstrap.cc:492] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: No bootstrap required, opened a new log
I20260924 00:50:30.348034 9097 ts_tablet_manager.cc:1397] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 00:50:30.347997 9098 log.cc:824] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:30.349294 9099 tablet_bootstrap.cc:492] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: No bootstrap required, opened a new log
I20260924 00:50:30.349555 9099 ts_tablet_manager.cc:1397] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 00:50:30.356822 9098 tablet_bootstrap.cc:492] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: No bootstrap required, opened a new log
I20260924 00:50:30.357160 9098 ts_tablet_manager.cc:1397] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260924 00:50:30.357453 9099 raft_consensus.cc:348] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.357834 9099 raft_consensus.cc:374] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:30.357929 9099 raft_consensus.cc:729] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f2d969822848c98c8fb04a91d1c2b0, State: Initialized, Role: FOLLOWER
I20260924 00:50:30.358033 9097 raft_consensus.cc:348] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.358481 9097 raft_consensus.cc:374] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:30.358601 9097 raft_consensus.cc:729] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df7ecfb4ee143c6b7f69865d39d14ef, State: Initialized, Role: FOLLOWER
I20260924 00:50:30.358575 9099 consensus_queue.cc:260] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.359282 9097 consensus_queue.cc:260] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.360674 9099 ts_tablet_manager.cc:1428] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260924 00:50:30.361917 8930 heartbeater.cc:503] Master 127.0.222.62:41957 was elected leader, sending a full tablet report...
I20260924 00:50:30.362603 9097 ts_tablet_manager.cc:1428] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260924 00:50:30.367359 9098 raft_consensus.cc:348] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.367832 9098 raft_consensus.cc:374] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:30.367941 9098 raft_consensus.cc:729] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec43e620c64ca0aba75da41c21b190, State: Initialized, Role: FOLLOWER
I20260924 00:50:30.368579 9098 consensus_queue.cc:260] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.369828 9072 heartbeater.cc:503] Master 127.0.222.62:41957 was elected leader, sending a full tablet report...
I20260924 00:50:30.370240 9098 ts_tablet_manager.cc:1428] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260924 00:50:30.386795 8791 tablet.cc:2369] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:50:30.416610 8931 tablet.cc:2369] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:50:30.457453 9073 tablet.cc:2369] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:50:30.634555 9105 raft_consensus.cc:482] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:30.634908 9105 raft_consensus.cc:504] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.636562 9105 leader_election.cc:288] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283), 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361)
I20260924 00:50:30.642074 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.637056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:37619 (local address 127.0.222.2:33283)
0924 00:50:30.637283 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.637290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.637311 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:50:30.637394 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.637399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.637471 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.637573 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:30.638014 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.638730 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.640288 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.641141 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.641294 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.641403 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.641674 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.641819 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:50:30.642105 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.636772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33283 (local address 127.0.222.3:37619)
0924 00:50:30.636988 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.637007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.637134 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.637610 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.637614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.637652 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:30.637885 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.637894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.638865 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.638869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.640119 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.640131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.641304 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.641309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.641512 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.641538 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.641648 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.641789 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"spinlock_wait_cycles":35712}
I20260924 00:50:30.642740 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.637057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47435 (local address 127.0.222.1:33361)
0924 00:50:30.637282 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.637287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.637636 (+ 349us) server_negotiation.cc:407] Connection header received
0924 00:50:30.637918 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.637923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.637998 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.638089 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:30.638613 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.639301 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.640938 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.641811 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.641928 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.642029 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.642359 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.642489 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:50:30.643266 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.636963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33361 (local address 127.0.222.3:47435)
0924 00:50:30.637548 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.637563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.637751 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.638126 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.638130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.638155 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:30.638472 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.638482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.639443 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.639446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.640794 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.640806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.642024 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.642030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.642171 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.642203 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.642911 (+ 708us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.643033 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":395,"threads_started":1}
I20260924 00:50:30.643293 8880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075f1d24b634ad38e63f2c38474790d" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" is_pre_election: true
I20260924 00:50:30.643913 8880 raft_consensus.cc:2454] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 0.
I20260924 00:50:30.644243 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075f1d24b634ad38e63f2c38474790d" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f2d969822848c98c8fb04a91d1c2b0" is_pre_election: true
I20260924 00:50:30.644613 8951 leader_election.cc:302] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec43e620c64ca0aba75da41c21b190, 2df7ecfb4ee143c6b7f69865d39d14ef; no voters:
I20260924 00:50:30.644801 8740 raft_consensus.cc:2454] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 0.
I20260924 00:50:30.645102 9105 raft_consensus.cc:2781] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:30.645291 9105 raft_consensus.cc:482] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:30.645402 9105 raft_consensus.cc:3037] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:30.647670 9105 raft_consensus.cc:504] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.648304 9105 leader_election.cc:288] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 election: Requested vote from peers 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283), 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361)
I20260924 00:50:30.648927 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075f1d24b634ad38e63f2c38474790d" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f2d969822848c98c8fb04a91d1c2b0"
I20260924 00:50:30.648927 8880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075f1d24b634ad38e63f2c38474790d" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
I20260924 00:50:30.649154 8880 raft_consensus.cc:3037] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:30.649154 8740 raft_consensus.cc:3037] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:30.652565 8880 raft_consensus.cc:2454] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 1.
I20260924 00:50:30.652565 8740 raft_consensus.cc:2454] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 1.
I20260924 00:50:30.653126 8950 leader_election.cc:302] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec43e620c64ca0aba75da41c21b190, 90f2d969822848c98c8fb04a91d1c2b0; no voters:
I20260924 00:50:30.653573 9105 raft_consensus.cc:2781] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:30.654126 9105 raft_consensus.cc:686] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 1 LEADER]: Becoming Leader. State: Replica: 1bec43e620c64ca0aba75da41c21b190, State: Running, Role: LEADER
I20260924 00:50:30.654662 9105 consensus_queue.cc:237] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } }
I20260924 00:50:30.659879 8581 catalog_manager.cc:5495] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "1bec43e620c64ca0aba75da41c21b190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } health_report { overall_health: UNKNOWN } } }
I20260924 00:50:30.688375 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.683022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.0.222.3:37627)
0924 00:50:30.683214 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.683220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.683237 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:50:30.683458 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.683462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.683550 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.683681 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:30.684196 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.684926 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.686312 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.686712 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.686825 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.687255 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:30.687799 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:30.687932 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.688021 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.688106 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:50:30.696856 8880 raft_consensus.cc:1261] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Refusing update from remote peer 1bec43e620c64ca0aba75da41c21b190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:30.696872 8740 raft_consensus.cc:1261] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Refusing update from remote peer 1bec43e620c64ca0aba75da41c21b190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:30.698170 9105 consensus_queue.cc:1035] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:30.698540 9107 consensus_queue.cc:1035] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:30.713452 9110 mvcc.cc:205] Tried to move back new op lower bound from 7332704381724086272 to 7332704381565591552. Current Snapshot: MvccSnapshot[applied={T|T < 7332704381724086272 or (T in {7332704381724086272})}]
I20260924 00:50:30.724416 8580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:30.724757 8580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:30.728772 8580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: Sending DeleteTablet for 3 replicas of tablet 5075f1d24b634ad38e63f2c38474790d
I20260924 00:50:30.730083 8860 tablet_service.cc:1514] Processing DeleteTablet for tablet 5075f1d24b634ad38e63f2c38474790d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:30 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:55334
I20260924 00:50:30.730098 9001 tablet_service.cc:1514] Processing DeleteTablet for tablet 5075f1d24b634ad38e63f2c38474790d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:30 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:50188
I20260924 00:50:30.730876 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet 5075f1d24b634ad38e63f2c38474790d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:30 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49152
I20260924 00:50:30.730984 9121 tablet_replica.cc:330] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: stopping tablet replica
I20260924 00:50:30.731572 9121 raft_consensus.cc:2229] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:30.732160 9121 raft_consensus.cc:2258] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:30.733974 8580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 00:50:30.734781 9121 ts_tablet_manager.cc:1905] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:50:30.735438 8580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:30.742087 9121 ts_tablet_manager.cc:1918] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:30.742446 9121 log.cc:1196] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/5075f1d24b634ad38e63f2c38474790d
I20260924 00:50:30.743080 9121 ts_tablet_manager.cc:1939] T 5075f1d24b634ad38e63f2c38474790d P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting consensus metadata
I20260924 00:50:30.745642 8566 catalog_manager.cc:4841] TS 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283): tablet 5075f1d24b634ad38e63f2c38474790d (table test-table [id=6f5039aa904449c188c19b8081b261ca]) successfully deleted
I20260924 00:50:30.745628 8719 tablet_service.cc:1467] Processing CreateTablet for tablet bf7546d413934bcd9c3b1981ba0c8559 (DEFAULT_TABLE table=test-table [id=1b09bb747e644b4382655f27e37bccf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:30.745685 8860 tablet_service.cc:1467] Processing CreateTablet for tablet bf7546d413934bcd9c3b1981ba0c8559 (DEFAULT_TABLE table=test-table [id=1b09bb747e644b4382655f27e37bccf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:30.746125 8860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7546d413934bcd9c3b1981ba0c8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:30.746222 8719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7546d413934bcd9c3b1981ba0c8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:30.749820 9000 tablet_service.cc:1467] Processing CreateTablet for tablet bf7546d413934bcd9c3b1981ba0c8559 (DEFAULT_TABLE table=test-table [id=1b09bb747e644b4382655f27e37bccf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:30.750487 9000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7546d413934bcd9c3b1981ba0c8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:30.753511 9099 tablet_bootstrap.cc:492] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: Bootstrap starting.
I20260924 00:50:30.753777 9123 tablet_replica.cc:330] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: stopping tablet replica
I20260924 00:50:30.754303 9123 raft_consensus.cc:2229] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:30.754876 9123 raft_consensus.cc:2258] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:30.755291 9099 tablet_bootstrap.cc:654] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:30.755774 9122 tablet_replica.cc:330] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: stopping tablet replica
I20260924 00:50:30.756230 9122 raft_consensus.cc:2229] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:50:30.757285 9122 raft_consensus.cc:2258] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:30.758553 9123 ts_tablet_manager.cc:1905] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:30.759174 9122 ts_tablet_manager.cc:1905] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:30.766642 9122 ts_tablet_manager.cc:1918] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:30.766921 9122 log.cc:1196] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/5075f1d24b634ad38e63f2c38474790d
I20260924 00:50:30.767522 9122 ts_tablet_manager.cc:1939] T 5075f1d24b634ad38e63f2c38474790d P 1bec43e620c64ca0aba75da41c21b190: Deleting consensus metadata
I20260924 00:50:30.768978 9098 tablet_bootstrap.cc:492] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: Bootstrap starting.
I20260924 00:50:30.769155 9097 tablet_bootstrap.cc:492] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: Bootstrap starting.
I20260924 00:50:30.770759 9098 tablet_bootstrap.cc:654] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:30.770797 9097 tablet_bootstrap.cc:654] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:30.773768 9099 tablet_bootstrap.cc:492] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: No bootstrap required, opened a new log
I20260924 00:50:30.773941 9099 ts_tablet_manager.cc:1397] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.002s
I20260924 00:50:30.774778 9099 raft_consensus.cc:348] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.775027 9099 raft_consensus.cc:374] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:30.775132 9099 raft_consensus.cc:729] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f2d969822848c98c8fb04a91d1c2b0, State: Initialized, Role: FOLLOWER
I20260924 00:50:30.775442 9099 consensus_queue.cc:260] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.778047 8566 catalog_manager.cc:4841] TS 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627): tablet 5075f1d24b634ad38e63f2c38474790d (table test-table [id=6f5039aa904449c188c19b8081b261ca]) successfully deleted
I20260924 00:50:30.778555 9099 ts_tablet_manager.cc:1428] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 00:50:30.781038 9123 ts_tablet_manager.cc:1918] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:30.781236 9123 log.cc:1196] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/5075f1d24b634ad38e63f2c38474790d
I20260924 00:50:30.781868 9123 ts_tablet_manager.cc:1939] T 5075f1d24b634ad38e63f2c38474790d P 90f2d969822848c98c8fb04a91d1c2b0: Deleting consensus metadata
I20260924 00:50:30.783274 9098 tablet_bootstrap.cc:492] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: No bootstrap required, opened a new log
I20260924 00:50:30.783454 9098 ts_tablet_manager.cc:1397] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260924 00:50:30.784060 8565 catalog_manager.cc:4841] TS 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361): tablet 5075f1d24b634ad38e63f2c38474790d (table test-table [id=6f5039aa904449c188c19b8081b261ca]) successfully deleted
I20260924 00:50:30.784565 9098 raft_consensus.cc:348] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.784771 9097 tablet_bootstrap.cc:492] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: No bootstrap required, opened a new log
I20260924 00:50:30.784825 9098 raft_consensus.cc:374] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:30.784929 9097 ts_tablet_manager.cc:1397] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20260924 00:50:30.784937 9098 raft_consensus.cc:729] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec43e620c64ca0aba75da41c21b190, State: Initialized, Role: FOLLOWER
I20260924 00:50:30.785231 9098 consensus_queue.cc:260] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.785851 9098 ts_tablet_manager.cc:1428] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 00:50:30.785784 9097 raft_consensus.cc:348] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.785997 9097 raft_consensus.cc:374] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:30.786079 9097 raft_consensus.cc:729] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df7ecfb4ee143c6b7f69865d39d14ef, State: Initialized, Role: FOLLOWER
I20260924 00:50:30.786326 9097 consensus_queue.cc:260] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.786886 9097 ts_tablet_manager.cc:1428] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:50:30.898149 9103 raft_consensus.cc:482] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:30.898398 9103 raft_consensus.cc:504] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.899714 9115 raft_consensus.cc:482] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:30.899963 9115 raft_consensus.cc:504] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.900300 9103 leader_election.cc:288] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283), 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627)
I20260924 00:50:30.900902 9115 leader_election.cc:288] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361), 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283)
I20260924 00:50:30.901865 8880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7546d413934bcd9c3b1981ba0c8559" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" is_pre_election: true
I20260924 00:50:30.902132 8880 raft_consensus.cc:2454] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 0.
I20260924 00:50:30.902947 8951 leader_election.cc:302] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec43e620c64ca0aba75da41c21b190, 2df7ecfb4ee143c6b7f69865d39d14ef; no voters:
I20260924 00:50:30.903252 9115 raft_consensus.cc:2781] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:30.903373 9115 raft_consensus.cc:482] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:30.903471 9115 raft_consensus.cc:3037] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:30.906740 9115 raft_consensus.cc:504] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.908190 8880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7546d413934bcd9c3b1981ba0c8559" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
I20260924 00:50:30.908434 8880 raft_consensus.cc:3037] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:30.908933 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.900741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:44409 (local address 127.0.222.2:33283)
0924 00:50:30.900930 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.900936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.901495 (+ 559us) server_negotiation.cc:407] Connection header received
0924 00:50:30.901773 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.901778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.901844 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.901913 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:30.902754 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.903876 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.906446 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.907635 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.907747 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.907846 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.908504 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.908641 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:50:30.909701 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.900521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:33283 (local address 127.0.222.1:44409)
0924 00:50:30.901397 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.901418 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.901603 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.902134 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.902139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.902167 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:30.902557 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.902572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.904060 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.904065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.906260 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.906277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.908069 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.908075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.908302 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.908338 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.909178 (+ 840us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.909352 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":554,"threads_started":1}
I20260924 00:50:30.911455 8880 raft_consensus.cc:2454] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 1.
I20260924 00:50:30.912009 8951 leader_election.cc:302] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec43e620c64ca0aba75da41c21b190, 2df7ecfb4ee143c6b7f69865d39d14ef; no voters:
I20260924 00:50:30.912369 9107 raft_consensus.cc:2781] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:30.912544 9107 raft_consensus.cc:686] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 1 LEADER]: Becoming Leader. State: Replica: 1bec43e620c64ca0aba75da41c21b190, State: Running, Role: LEADER
I20260924 00:50:30.912848 9107 consensus_queue.cc:237] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:30.914196 8740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7546d413934bcd9c3b1981ba0c8559" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f2d969822848c98c8fb04a91d1c2b0" is_pre_election: true
I20260924 00:50:30.914450 8740 raft_consensus.cc:2454] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 0.
I20260924 00:50:30.914873 8879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7546d413934bcd9c3b1981ba0c8559" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" is_pre_election: true
I20260924 00:50:30.915241 8879 raft_consensus.cc:2379] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90f2d969822848c98c8fb04a91d1c2b0 in current term 1: Already voted for candidate 1bec43e620c64ca0aba75da41c21b190 in this term.
I20260924 00:50:30.915710 8739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7546d413934bcd9c3b1981ba0c8559" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f2d969822848c98c8fb04a91d1c2b0"
I20260924 00:50:30.916013 8739 raft_consensus.cc:3037] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:30.916317 9115 leader_election.cc:288] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 election: Requested vote from peers 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361), 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283)
I20260924 00:50:30.916786 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.902487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:34563 (local address 127.0.222.3:37627)
0924 00:50:30.902659 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.902665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.910052 (+ 7387us) server_negotiation.cc:407] Connection header received
0924 00:50:30.910308 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.910314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.910415 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.910481 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:50:30.911038 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.912091 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.914366 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.915523 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.915644 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.915762 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:30.916356 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 00:50:30.916514 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:50:30.916829 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.901546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:37627 (local address 127.0.222.1:34563)
0924 00:50:30.909959 (+ 8413us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.909979 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.910159 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.910694 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.910699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.910726 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:50:30.910894 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.910904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.912268 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.912273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.914153 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.914177 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.915933 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.915939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.916154 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.916192 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:50:30.916294 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 00:50:30.916478 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8278,"thread_start_us":12055,"threads_started":1}
I20260924 00:50:30.917971 9025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7546d413934bcd9c3b1981ba0c8559" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec43e620c64ca0aba75da41c21b190" is_pre_election: true
I20260924 00:50:30.919265 8670 leader_election.cc:302] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90f2d969822848c98c8fb04a91d1c2b0; no voters: 1bec43e620c64ca0aba75da41c21b190, 2df7ecfb4ee143c6b7f69865d39d14ef
I20260924 00:50:30.919448 8739 raft_consensus.cc:2454] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 1.
I20260924 00:50:30.919730 9103 raft_consensus.cc:2726] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 00:50:30.919517 8580 catalog_manager.cc:5495] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "1bec43e620c64ca0aba75da41c21b190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } health_report { overall_health: HEALTHY } } }
Sep 24 00:50:30 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211028, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.62@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:31.001850 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.987756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41957 (local address 127.0.0.1:37494)
0924 00:50:30.988533 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:30.988548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:30.988696 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:30.989046 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:30.989054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:30.989726 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:30.989916 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.989927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.990929 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.990934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.991865 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:30.991877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.992570 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.992576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:30.992734 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.993483 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:30.993501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:30.996164 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:30.999236 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:30.999249 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:30.999258 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:30.999666 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:31.000002 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:31.000014 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:31.000018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:31.000153 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:31.000692 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:31.000700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:31.000866 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 00:50:31.001025 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 00:50:31.001333 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":537,"threads_started":1}
I20260924 00:50:31.001871 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:30.987761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.0.222.62:41957)
0924 00:50:30.988403 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:30.988409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:30.988635 (+ 226us) server_negotiation.cc:407] Connection header received
0924 00:50:30.988809 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:30.988814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:30.988879 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:30.988968 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:30.990051 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.990792 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.991992 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:30.992416 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:30.992522 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:30.996352 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:30.996357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:30.996360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:30.996390 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:30.999102 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:30.999783 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:30.999789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:30.999791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:30.999906 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:31.000287 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:31.000292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:31.000295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:31.000571 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:31.000698 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:31.001126 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 00:50:31.001468 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":475,"thread_start_us":308,"threads_started":1}
I20260924 00:50:31.007352 8580 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:37494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:50:31.008467 8580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:31.016048 8720 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9ef90d3c4b4b06b0e55f6598cea28a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f32e636b2c94f1ab3a5c8bffc246445]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:50:31.016155 8860 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9ef90d3c4b4b06b0e55f6598cea28a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f32e636b2c94f1ab3a5c8bffc246445]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:50:31.016698 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9ef90d3c4b4b06b0e55f6598cea28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.016698 8860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9ef90d3c4b4b06b0e55f6598cea28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.016615 9000 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9ef90d3c4b4b06b0e55f6598cea28a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f32e636b2c94f1ab3a5c8bffc246445]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:50:31.017144 9000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9ef90d3c4b4b06b0e55f6598cea28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.023854 9099 tablet_bootstrap.cc:492] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: Bootstrap starting.
I20260924 00:50:31.023854 9097 tablet_bootstrap.cc:492] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: Bootstrap starting.
I20260924 00:50:31.025640 9097 tablet_bootstrap.cc:654] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.025699 9099 tablet_bootstrap.cc:654] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.026423 9098 tablet_bootstrap.cc:492] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: Bootstrap starting.
I20260924 00:50:31.028250 9098 tablet_bootstrap.cc:654] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.035322 9098 tablet_bootstrap.cc:492] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: No bootstrap required, opened a new log
I20260924 00:50:31.035728 9098 ts_tablet_manager.cc:1397] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260924 00:50:31.036653 9098 raft_consensus.cc:348] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.037078 9098 raft_consensus.cc:374] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.037673 9098 raft_consensus.cc:729] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec43e620c64ca0aba75da41c21b190, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.038064 9098 consensus_queue.cc:260] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.038862 9107 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.039043 9099 tablet_bootstrap.cc:492] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: No bootstrap required, opened a new log
I20260924 00:50:31.039124 9097 tablet_bootstrap.cc:492] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: No bootstrap required, opened a new log
I20260924 00:50:31.039146 9107 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:50:31.039232 9099 ts_tablet_manager.cc:1397] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260924 00:50:31.039273 9097 ts_tablet_manager.cc:1397] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260924 00:50:31.040161 9097 raft_consensus.cc:348] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.040416 9097 raft_consensus.cc:374] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.040506 9097 raft_consensus.cc:729] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df7ecfb4ee143c6b7f69865d39d14ef, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.040802 9097 consensus_queue.cc:260] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.041483 9097 ts_tablet_manager.cc:1428] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 00:50:31.041596 9104 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.041988 9104 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:50:31.042078 9099 raft_consensus.cc:348] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.042366 9099 raft_consensus.cc:374] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.042423 9099 raft_consensus.cc:729] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f2d969822848c98c8fb04a91d1c2b0, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.042637 9099 consensus_queue.cc:260] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.043169 9103 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.043495 9103 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:50:31.044155 9099 ts_tablet_manager.cc:1428] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 00:50:31.045128 9098 ts_tablet_manager.cc:1428] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260924 00:50:31.111284 9103 raft_consensus.cc:482] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:31.111379 9107 raft_consensus.cc:482] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:31.111500 9103 raft_consensus.cc:504] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.111581 9107 raft_consensus.cc:504] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.112157 9103 leader_election.cc:288] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283), 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627)
I20260924 00:50:31.112260 9107 leader_election.cc:288] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361), 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283)
I20260924 00:50:31.112807 8879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9ef90d3c4b4b06b0e55f6598cea28a" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" is_pre_election: true
I20260924 00:50:31.112950 9025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9ef90d3c4b4b06b0e55f6598cea28a" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec43e620c64ca0aba75da41c21b190" is_pre_election: true
I20260924 00:50:31.112975 8739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9ef90d3c4b4b06b0e55f6598cea28a" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f2d969822848c98c8fb04a91d1c2b0" is_pre_election: true
I20260924 00:50:31.113056 8879 raft_consensus.cc:2454] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f2d969822848c98c8fb04a91d1c2b0 in term 0.
I20260924 00:50:31.113250 9025 raft_consensus.cc:2454] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f2d969822848c98c8fb04a91d1c2b0 in term 0.
I20260924 00:50:31.113277 8739 raft_consensus.cc:2454] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 0.
I20260924 00:50:31.113685 8670 leader_election.cc:302] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df7ecfb4ee143c6b7f69865d39d14ef, 90f2d969822848c98c8fb04a91d1c2b0; no voters:
I20260924 00:50:31.113816 8879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9ef90d3c4b4b06b0e55f6598cea28a" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" is_pre_election: true
I20260924 00:50:31.113902 8950 leader_election.cc:302] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec43e620c64ca0aba75da41c21b190, 90f2d969822848c98c8fb04a91d1c2b0; no voters:
I20260924 00:50:31.113993 8879 raft_consensus.cc:2454] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 0.
I20260924 00:50:31.114132 9103 raft_consensus.cc:2781] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:31.114266 9103 raft_consensus.cc:482] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:31.114274 9107 raft_consensus.cc:2781] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:31.114358 9103 raft_consensus.cc:3037] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.114393 9107 raft_consensus.cc:482] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:31.114495 9107 raft_consensus.cc:3037] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.116460 9103 raft_consensus.cc:504] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.116940 9103 leader_election.cc:288] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 election: Requested vote from peers 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283), 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627)
I20260924 00:50:31.117455 8879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9ef90d3c4b4b06b0e55f6598cea28a" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
I20260924 00:50:31.117486 9025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9ef90d3c4b4b06b0e55f6598cea28a" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec43e620c64ca0aba75da41c21b190"
I20260924 00:50:31.117627 8879 raft_consensus.cc:3037] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.117605 9107 raft_consensus.cc:504] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.118074 9025 raft_consensus.cc:2379] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 90f2d969822848c98c8fb04a91d1c2b0 in current term 1: Already voted for candidate 1bec43e620c64ca0aba75da41c21b190 in this term.
I20260924 00:50:31.118655 8739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9ef90d3c4b4b06b0e55f6598cea28a" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f2d969822848c98c8fb04a91d1c2b0"
I20260924 00:50:31.118806 9107 leader_election.cc:288] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 election: Requested vote from peers 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361), 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283)
I20260924 00:50:31.118933 8739 raft_consensus.cc:2379] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bec43e620c64ca0aba75da41c21b190 in current term 1: Already voted for candidate 90f2d969822848c98c8fb04a91d1c2b0 in this term.
I20260924 00:50:31.119519 8880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9ef90d3c4b4b06b0e55f6598cea28a" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
I20260924 00:50:31.120146 8951 leader_election.cc:302] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bec43e620c64ca0aba75da41c21b190; no voters: 2df7ecfb4ee143c6b7f69865d39d14ef, 90f2d969822848c98c8fb04a91d1c2b0
I20260924 00:50:31.120468 8879 raft_consensus.cc:2454] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f2d969822848c98c8fb04a91d1c2b0 in term 1.
I20260924 00:50:31.120612 9107 raft_consensus.cc:2726] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 00:50:31.120872 8670 leader_election.cc:302] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2df7ecfb4ee143c6b7f69865d39d14ef, 90f2d969822848c98c8fb04a91d1c2b0; no voters: 1bec43e620c64ca0aba75da41c21b190
I20260924 00:50:31.121178 9103 raft_consensus.cc:2781] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:31.121665 9103 raft_consensus.cc:686] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 LEADER]: Becoming Leader. State: Replica: 90f2d969822848c98c8fb04a91d1c2b0, State: Running, Role: LEADER
I20260924 00:50:31.122375 9103 consensus_queue.cc:237] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.124969 9140 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: TxnStatusTablet state changed. Reason: New leader 90f2d969822848c98c8fb04a91d1c2b0. Latest consensus state: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.125444 9140 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:50:31.126260 9142 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:50:31.126510 9142 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:50:31.126597 8580 catalog_manager.cc:5495] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } health_report { overall_health: UNKNOWN } } }
I20260924 00:50:31.172981 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.160306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.0.222.1:33361)
0924 00:50:31.165717 (+ 5411us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:31.165723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:31.165748 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:50:31.165854 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:31.165859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:31.165967 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:31.166147 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:31.166683 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.167983 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.170040 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.170748 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.170878 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.171453 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.172245 (+ 792us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.172377 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:31.172533 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 00:50:31.172628 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5259,"thread_start_us":5097,"threads_started":1}
I20260924 00:50:31.173002 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.159256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33361 (local address 127.0.0.1:49162)
0924 00:50:31.159811 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:31.159832 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:31.159985 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:31.166264 (+ 6279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:31.166271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:31.166288 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:50:31.166515 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:31.166527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.168159 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.168165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:31.169875 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:31.169894 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.170926 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.170933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:31.171138 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.171240 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.172365 (+ 1125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.172408 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 00:50:31.172502 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 00:50:31.172673 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354}
I20260924 00:50:31.178149 9025 raft_consensus.cc:1261] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Refusing update from remote peer 90f2d969822848c98c8fb04a91d1c2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:31.178270 8879 raft_consensus.cc:1261] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Refusing update from remote peer 90f2d969822848c98c8fb04a91d1c2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:31.178912 9103 consensus_queue.cc:1035] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:31.179643 9140 consensus_queue.cc:1035] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:31.193665 9107 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: TxnStatusTablet state changed. Reason: New leader 90f2d969822848c98c8fb04a91d1c2b0. Latest consensus state: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.193976 9107 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:50:31.195392 9104 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: TxnStatusTablet state changed. Reason: New leader 90f2d969822848c98c8fb04a91d1c2b0. Latest consensus state: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.195762 9104 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:50:31.196743 9111 mvcc.cc:205] Tried to move back new op lower bound from 7332704383696482304 to 7332704383481884672. Current Snapshot: MvccSnapshot[applied={T|T < 7332704383696482304}]
I20260924 00:50:31.197590 9107 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.197916 9107 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 1bec43e620c64ca0aba75da41c21b190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:50:31.198529 9103 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.198839 9103 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:50:31.204063 9140 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.204362 9140 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 90f2d969822848c98c8fb04a91d1c2b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:50:31.205555 9104 tablet_replica.cc:439] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } } }
I20260924 00:50:31.205837 9104 tablet_replica.cc:442] T 3b9ef90d3c4b4b06b0e55f6598cea28a P 2df7ecfb4ee143c6b7f69865d39d14ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:50:31.224184 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.217799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:51267 (local address 127.0.222.1:33361)
0924 00:50:31.217990 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:31.217996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:31.218717 (+ 721us) server_negotiation.cc:407] Connection header received
0924 00:50:31.218951 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:31.218958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:31.219040 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:31.219139 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:31.219689 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.220680 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.222259 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.222718 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.222856 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.223336 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.223464 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.223549 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:31.223730 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 00:50:31.223817 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:50:31.224272 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.217786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33361 (local address 127.0.222.3:51267)
0924 00:50:31.218590 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:31.218617 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:31.218794 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:31.219167 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:31.219171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:31.219184 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:50:31.219520 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:31.219531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.220835 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.220840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:31.222096 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:31.222114 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.222933 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.222940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:31.223081 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.223153 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.223565 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.223579 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 00:50:31.223672 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 00:50:31.223826 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":418,"threads_started":1}
I20260924 00:50:31.237809 8739 raft_consensus.cc:1261] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Refusing update from remote peer 1bec43e620c64ca0aba75da41c21b190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:31.237815 8879 raft_consensus.cc:1261] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Refusing update from remote peer 1bec43e620c64ca0aba75da41c21b190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:31.238389 9115 consensus_queue.cc:1035] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:31.238745 9107 consensus_queue.cc:1035] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:31.284915 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.277806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:37933 (local address 127.0.222.3:37627)
0924 00:50:31.277961 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:31.277966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:31.278533 (+ 567us) server_negotiation.cc:407] Connection header received
0924 00:50:31.278753 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:31.278757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:31.278825 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:31.278898 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:31.279523 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.280632 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.282671 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.283303 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.283429 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.284105 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.284258 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.284321 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:31.284529 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 00:50:31.284604 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:50:31.285168 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.277737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:37627 (local address 127.0.222.1:37933)
0924 00:50:31.278447 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:31.278468 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:31.278605 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:31.278930 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:31.278935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:31.278946 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:50:31.279360 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:31.279373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.280822 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.280829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:31.282534 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:31.282551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.283589 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.283596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:31.283788 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.283868 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.284398 (+ 530us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.284408 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 00:50:31.284485 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:50:31.284638 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":365,"threads_started":1}
I20260924 00:50:31.319386 8580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:31.319635 8580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:31.322779 8580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: Sending DeleteTablet for 3 replicas of tablet bf7546d413934bcd9c3b1981ba0c8559
I20260924 00:50:31.323748 8860 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7546d413934bcd9c3b1981ba0c8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:31 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:55334
I20260924 00:50:31.323832 9000 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7546d413934bcd9c3b1981ba0c8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:31 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:50188
I20260924 00:50:31.326027 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet bf7546d413934bcd9c3b1981ba0c8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:31 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49152
I20260924 00:50:31.339401 9157 tablet_replica.cc:330] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: stopping tablet replica
I20260924 00:50:31.339679 9157 raft_consensus.cc:2229] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:31.339989 9157 raft_consensus.cc:2258] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:31.341398 9157 ts_tablet_manager.cc:1905] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:31.347675 9159 tablet_replica.cc:330] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: stopping tablet replica
I20260924 00:50:31.348788 9157 ts_tablet_manager.cc:1918] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:50:31.348938 9157 log.cc:1196] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/bf7546d413934bcd9c3b1981ba0c8559
I20260924 00:50:31.349359 9157 ts_tablet_manager.cc:1939] T bf7546d413934bcd9c3b1981ba0c8559 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting consensus metadata
I20260924 00:50:31.350454 8566 catalog_manager.cc:4841] TS 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283): tablet bf7546d413934bcd9c3b1981ba0c8559 (table test-table [id=1b09bb747e644b4382655f27e37bccf3]) successfully deleted
I20260924 00:50:31.350553 9159 raft_consensus.cc:2229] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:31.350921 9159 raft_consensus.cc:2258] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:31.351480 9158 tablet_replica.cc:330] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: stopping tablet replica
I20260924 00:50:31.351683 9158 raft_consensus.cc:2229] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:50:31.352137 9158 raft_consensus.cc:2258] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:31.352212 9159 ts_tablet_manager.cc:1905] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:31.353300 9158 ts_tablet_manager.cc:1905] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:31.359830 9159 ts_tablet_manager.cc:1918] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:50:31.360014 9159 log.cc:1196] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/bf7546d413934bcd9c3b1981ba0c8559
I20260924 00:50:31.360462 9159 ts_tablet_manager.cc:1939] T bf7546d413934bcd9c3b1981ba0c8559 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting consensus metadata
I20260924 00:50:31.360813 9158 ts_tablet_manager.cc:1918] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:50:31.360992 9158 log.cc:1196] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/bf7546d413934bcd9c3b1981ba0c8559
I20260924 00:50:31.361488 9158 ts_tablet_manager.cc:1939] T bf7546d413934bcd9c3b1981ba0c8559 P 1bec43e620c64ca0aba75da41c21b190: Deleting consensus metadata
I20260924 00:50:31.361649 8565 catalog_manager.cc:4841] TS 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361): tablet bf7546d413934bcd9c3b1981ba0c8559 (table test-table [id=1b09bb747e644b4382655f27e37bccf3]) successfully deleted
I20260924 00:50:31.362582 8566 catalog_manager.cc:4841] TS 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627): tablet bf7546d413934bcd9c3b1981ba0c8559 (table test-table [id=1b09bb747e644b4382655f27e37bccf3]) successfully deleted
Sep 24 00:50:31 dist-test-slave-rtvz krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:50:31 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:31 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:50:31.399361 9135 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:31.386974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.0.222.62:41957)
0924 00:50:31.387229 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:31.387235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:31.387683 (+ 448us) server_negotiation.cc:407] Connection header received
0924 00:50:31.387789 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:31.387793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:31.387865 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:31.387963 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:31.389803 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.390527 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.392336 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.392838 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.392950 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.399093 (+ 6143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37510: BlockingRecv error: recv got EOF from 127.0.0.1:37510 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 00:50:31 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM
I20260924 00:50:31.419080 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.402934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.0.222.62:41957)
0924 00:50:31.403149 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:31.403157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:31.403629 (+ 472us) server_negotiation.cc:407] Connection header received
0924 00:50:31.403804 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:31.403810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:31.403898 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:31.404011 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:31.405366 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.406500 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.407890 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.408561 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.408684 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.412887 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:31.412893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:31.412898 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:31.412943 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:31.416261 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:31.417031 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:31.417036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:31.417039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:31.417259 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:31.417617 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:31.417621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:31.417624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:31.417959 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:31.418213 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:31.418479 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 00:50:31.418716 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:50:31.422874 8580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:50:31.423491 8580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:31.430599 8720 tablet_service.cc:1467] Processing CreateTablet for tablet dd7a1afb69f0477bba1a37f92288c7f8 (DEFAULT_TABLE table=test-table [id=0161b95d54784a51b6a0a97425d4c61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:31.430869 8860 tablet_service.cc:1467] Processing CreateTablet for tablet dd7a1afb69f0477bba1a37f92288c7f8 (DEFAULT_TABLE table=test-table [id=0161b95d54784a51b6a0a97425d4c61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:31.430972 9000 tablet_service.cc:1467] Processing CreateTablet for tablet dd7a1afb69f0477bba1a37f92288c7f8 (DEFAULT_TABLE table=test-table [id=0161b95d54784a51b6a0a97425d4c61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:31.431109 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7a1afb69f0477bba1a37f92288c7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.431274 8860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7a1afb69f0477bba1a37f92288c7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.431447 9000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd7a1afb69f0477bba1a37f92288c7f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.437825 9097 tablet_bootstrap.cc:492] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: Bootstrap starting.
I20260924 00:50:31.437836 9099 tablet_bootstrap.cc:492] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: Bootstrap starting.
I20260924 00:50:31.439360 9098 tablet_bootstrap.cc:492] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: Bootstrap starting.
I20260924 00:50:31.439468 9099 tablet_bootstrap.cc:654] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.439469 9097 tablet_bootstrap.cc:654] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.440757 9098 tablet_bootstrap.cc:654] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.453805 9099 tablet_bootstrap.cc:492] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: No bootstrap required, opened a new log
I20260924 00:50:31.454008 9099 ts_tablet_manager.cc:1397] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260924 00:50:31.454978 9099 raft_consensus.cc:348] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.455327 9099 raft_consensus.cc:374] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.455446 9099 raft_consensus.cc:729] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f2d969822848c98c8fb04a91d1c2b0, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.455667 9098 tablet_bootstrap.cc:492] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: No bootstrap required, opened a new log
I20260924 00:50:31.455696 9099 consensus_queue.cc:260] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.455850 9098 ts_tablet_manager.cc:1397] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.013s
I20260924 00:50:31.456669 9098 raft_consensus.cc:348] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.456957 9098 raft_consensus.cc:374] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.457068 9098 raft_consensus.cc:729] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec43e620c64ca0aba75da41c21b190, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.458475 9098 consensus_queue.cc:260] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.458688 9097 tablet_bootstrap.cc:492] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: No bootstrap required, opened a new log
I20260924 00:50:31.458868 9097 ts_tablet_manager.cc:1397] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.014s
I20260924 00:50:31.459121 9098 ts_tablet_manager.cc:1428] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 00:50:31.459661 9097 raft_consensus.cc:348] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.459939 9097 raft_consensus.cc:374] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.460022 9099 ts_tablet_manager.cc:1428] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 00:50:31.460045 9097 raft_consensus.cc:729] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df7ecfb4ee143c6b7f69865d39d14ef, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.460314 9097 consensus_queue.cc:260] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.461344 9097 ts_tablet_manager.cc:1428] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 00:50:31.710091 9105 raft_consensus.cc:482] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:31.710290 9105 raft_consensus.cc:504] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.710912 9105 leader_election.cc:288] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361), 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283)
I20260924 00:50:31.711530 8739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7a1afb69f0477bba1a37f92288c7f8" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f2d969822848c98c8fb04a91d1c2b0" is_pre_election: true
I20260924 00:50:31.711575 8879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7a1afb69f0477bba1a37f92288c7f8" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" is_pre_election: true
I20260924 00:50:31.711750 8739 raft_consensus.cc:2454] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 0.
I20260924 00:50:31.711822 8879 raft_consensus.cc:2454] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 0.
I20260924 00:50:31.712268 8950 leader_election.cc:302] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec43e620c64ca0aba75da41c21b190, 90f2d969822848c98c8fb04a91d1c2b0; no voters:
I20260924 00:50:31.712599 9105 raft_consensus.cc:2781] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:31.712711 9105 raft_consensus.cc:482] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:31.712785 9105 raft_consensus.cc:3037] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.714828 9105 raft_consensus.cc:504] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.715337 9105 leader_election.cc:288] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 election: Requested vote from peers 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361), 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283)
I20260924 00:50:31.715839 8879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7a1afb69f0477bba1a37f92288c7f8" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
I20260924 00:50:31.715860 8739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd7a1afb69f0477bba1a37f92288c7f8" candidate_uuid: "1bec43e620c64ca0aba75da41c21b190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f2d969822848c98c8fb04a91d1c2b0"
I20260924 00:50:31.716038 8739 raft_consensus.cc:3037] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.716048 8879 raft_consensus.cc:3037] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.718181 8879 raft_consensus.cc:2454] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 1.
I20260924 00:50:31.718212 8739 raft_consensus.cc:2454] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bec43e620c64ca0aba75da41c21b190 in term 1.
I20260924 00:50:31.718662 8951 leader_election.cc:302] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec43e620c64ca0aba75da41c21b190, 2df7ecfb4ee143c6b7f69865d39d14ef; no voters:
I20260924 00:50:31.718982 9105 raft_consensus.cc:2781] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:31.719144 9105 raft_consensus.cc:686] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 1 LEADER]: Becoming Leader. State: Replica: 1bec43e620c64ca0aba75da41c21b190, State: Running, Role: LEADER
I20260924 00:50:31.719445 9105 consensus_queue.cc:237] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.722052 8580 catalog_manager.cc:5495] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "1bec43e620c64ca0aba75da41c21b190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } health_report { overall_health: HEALTHY } } }
I20260924 00:50:31.754846 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.750186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.0.222.3:37627)
0924 00:50:31.750414 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:31.750420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:31.750463 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:50:31.750568 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:31.750572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:31.750639 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:31.750711 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:31.751097 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.751860 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.753158 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.753630 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.753753 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.754152 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.754264 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.754339 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:31.754471 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 00:50:31.754555 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:50:31.757484 8739 raft_consensus.cc:1261] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Refusing update from remote peer 1bec43e620c64ca0aba75da41c21b190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:31.757516 8879 raft_consensus.cc:1261] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Refusing update from remote peer 1bec43e620c64ca0aba75da41c21b190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:31.758157 9115 consensus_queue.cc:1035] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:31.758411 9105 consensus_queue.cc:1035] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:31.768608 8581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:31.768925 8581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:31.772431 8581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: Sending DeleteTablet for 3 replicas of tablet dd7a1afb69f0477bba1a37f92288c7f8
I20260924 00:50:31.773396 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7a1afb69f0477bba1a37f92288c7f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:31 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49152
I20260924 00:50:31.773706 9159 tablet_replica.cc:330] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: stopping tablet replica
I20260924 00:50:31.773638 8860 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7a1afb69f0477bba1a37f92288c7f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:31 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:55334
I20260924 00:50:31.773715 8859 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7a1afb69f0477bba1a37f92288c7f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:31 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:55334
I20260924 00:50:31.773911 9159 raft_consensus.cc:2229] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:31.773972 9157 tablet_replica.cc:330] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: stopping tablet replica
I20260924 00:50:31.774143 9159 raft_consensus.cc:2258] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:31.774168 9157 raft_consensus.cc:2229] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:31.774255 9000 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7a1afb69f0477bba1a37f92288c7f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:31 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:50188
I20260924 00:50:31.774394 9157 raft_consensus.cc:2258] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:31.774425 9001 tablet_service.cc:1514] Processing DeleteTablet for tablet dd7a1afb69f0477bba1a37f92288c7f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:31 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:50188
I20260924 00:50:31.774641 9158 tablet_replica.cc:330] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: stopping tablet replica
I20260924 00:50:31.774866 9158 raft_consensus.cc:2229] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:50:31.775195 9158 raft_consensus.cc:2258] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:31.775429 9159 ts_tablet_manager.cc:1905] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:31.776024 9157 ts_tablet_manager.cc:1905] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:31.776120 9158 ts_tablet_manager.cc:1905] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:31.776502 8581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:50:31.777338 8581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:50:31.782436 9159 ts_tablet_manager.cc:1918] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:31.782506 9157 ts_tablet_manager.cc:1918] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:31.782585 9159 log.cc:1196] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/dd7a1afb69f0477bba1a37f92288c7f8
I20260924 00:50:31.782645 9157 log.cc:1196] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/dd7a1afb69f0477bba1a37f92288c7f8
I20260924 00:50:31.782963 9159 ts_tablet_manager.cc:1939] T dd7a1afb69f0477bba1a37f92288c7f8 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting consensus metadata
I20260924 00:50:31.783002 9157 ts_tablet_manager.cc:1939] T dd7a1afb69f0477bba1a37f92288c7f8 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting consensus metadata
I20260924 00:50:31.783351 9158 ts_tablet_manager.cc:1918] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:50:31.783500 9158 log.cc:1196] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/dd7a1afb69f0477bba1a37f92288c7f8
I20260924 00:50:31.784114 8565 catalog_manager.cc:4841] TS 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361): tablet dd7a1afb69f0477bba1a37f92288c7f8 (table test-table [id=0161b95d54784a51b6a0a97425d4c61b]) successfully deleted
I20260924 00:50:31.784142 8566 catalog_manager.cc:4841] TS 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283): tablet dd7a1afb69f0477bba1a37f92288c7f8 (table test-table [id=0161b95d54784a51b6a0a97425d4c61b]) successfully deleted
I20260924 00:50:31.784564 9158 ts_tablet_manager.cc:1939] T dd7a1afb69f0477bba1a37f92288c7f8 P 1bec43e620c64ca0aba75da41c21b190: Deleting consensus metadata
W20260924 00:50:31.784608 8566 catalog_manager.cc:4808] TS 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283): delete failed for tablet dd7a1afb69f0477bba1a37f92288c7f8 because the tablet was not found. No further retry: Not found: Tablet not found: dd7a1afb69f0477bba1a37f92288c7f8
I20260924 00:50:31.785772 8566 catalog_manager.cc:4841] TS 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627): tablet dd7a1afb69f0477bba1a37f92288c7f8 (table test-table [id=0161b95d54784a51b6a0a97425d4c61b]) successfully deleted
W20260924 00:50:31.786041 8566 catalog_manager.cc:4808] TS 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627): delete failed for tablet dd7a1afb69f0477bba1a37f92288c7f8 because the tablet was not found. No further retry: Not found: Tablet not found: dd7a1afb69f0477bba1a37f92288c7f8
I20260924 00:50:31.788548 8720 tablet_service.cc:1467] Processing CreateTablet for tablet f66796a34c3c4f94890daa8260da8c75 (DEFAULT_TABLE table=test-table [id=86de7ad0cb76472a901575b45b83c464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:31.788678 8859 tablet_service.cc:1467] Processing CreateTablet for tablet f66796a34c3c4f94890daa8260da8c75 (DEFAULT_TABLE table=test-table [id=86de7ad0cb76472a901575b45b83c464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:31.788834 9001 tablet_service.cc:1467] Processing CreateTablet for tablet f66796a34c3c4f94890daa8260da8c75 (DEFAULT_TABLE table=test-table [id=86de7ad0cb76472a901575b45b83c464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:50:31.789069 8720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f66796a34c3c4f94890daa8260da8c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.789116 8859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f66796a34c3c4f94890daa8260da8c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.789275 9001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f66796a34c3c4f94890daa8260da8c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:31.794137 9097 tablet_bootstrap.cc:492] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: Bootstrap starting.
I20260924 00:50:31.795362 9099 tablet_bootstrap.cc:492] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: Bootstrap starting.
I20260924 00:50:31.795415 9098 tablet_bootstrap.cc:492] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: Bootstrap starting.
I20260924 00:50:31.795674 9097 tablet_bootstrap.cc:654] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.796777 9098 tablet_bootstrap.cc:654] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.796952 9099 tablet_bootstrap.cc:654] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:31.798050 9097 tablet_bootstrap.cc:492] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: No bootstrap required, opened a new log
I20260924 00:50:31.798207 9097 ts_tablet_manager.cc:1397] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 00:50:31.798970 9099 tablet_bootstrap.cc:492] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: No bootstrap required, opened a new log
I20260924 00:50:31.798954 9097 raft_consensus.cc:348] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.799220 9099 ts_tablet_manager.cc:1397] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 00:50:31.799216 9097 raft_consensus.cc:374] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.799293 9097 raft_consensus.cc:729] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df7ecfb4ee143c6b7f69865d39d14ef, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.799538 9097 consensus_queue.cc:260] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.800025 9099 raft_consensus.cc:348] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.800223 9097 ts_tablet_manager.cc:1428] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 00:50:31.800279 9099 raft_consensus.cc:374] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.800379 9099 raft_consensus.cc:729] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f2d969822848c98c8fb04a91d1c2b0, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.800879 9099 consensus_queue.cc:260] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.801581 9098 tablet_bootstrap.cc:492] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: No bootstrap required, opened a new log
I20260924 00:50:31.801748 9098 ts_tablet_manager.cc:1397] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 00:50:31.802174 9099 ts_tablet_manager.cc:1428] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:31.802517 9098 raft_consensus.cc:348] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.802768 9098 raft_consensus.cc:374] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:31.802866 9098 raft_consensus.cc:729] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec43e620c64ca0aba75da41c21b190, State: Initialized, Role: FOLLOWER
I20260924 00:50:31.803102 9098 consensus_queue.cc:260] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.803583 9098 ts_tablet_manager.cc:1428] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 00:50:31.888175 9140 raft_consensus.cc:482] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:50:31.888422 9140 raft_consensus.cc:504] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.889086 9140 leader_election.cc:288] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283), 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627)
I20260924 00:50:31.889751 8879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66796a34c3c4f94890daa8260da8c75" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" is_pre_election: true
I20260924 00:50:31.889905 9025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66796a34c3c4f94890daa8260da8c75" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec43e620c64ca0aba75da41c21b190" is_pre_election: true
I20260924 00:50:31.890025 8879 raft_consensus.cc:2454] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f2d969822848c98c8fb04a91d1c2b0 in term 0.
I20260924 00:50:31.890136 9025 raft_consensus.cc:2454] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f2d969822848c98c8fb04a91d1c2b0 in term 0.
I20260924 00:50:31.890437 8670 leader_election.cc:302] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df7ecfb4ee143c6b7f69865d39d14ef, 90f2d969822848c98c8fb04a91d1c2b0; no voters:
I20260924 00:50:31.890719 9140 raft_consensus.cc:2781] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:50:31.890846 9140 raft_consensus.cc:482] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:50:31.890949 9140 raft_consensus.cc:3037] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.892959 9140 raft_consensus.cc:504] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.893625 9140 leader_election.cc:288] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 election: Requested vote from peers 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283), 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627)
I20260924 00:50:31.894271 8879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66796a34c3c4f94890daa8260da8c75" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef"
I20260924 00:50:31.894299 9025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f66796a34c3c4f94890daa8260da8c75" candidate_uuid: "90f2d969822848c98c8fb04a91d1c2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec43e620c64ca0aba75da41c21b190"
I20260924 00:50:31.894517 8879 raft_consensus.cc:3037] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.894531 9025 raft_consensus.cc:3037] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:31.896750 8879 raft_consensus.cc:2454] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f2d969822848c98c8fb04a91d1c2b0 in term 1.
I20260924 00:50:31.896801 9025 raft_consensus.cc:2454] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90f2d969822848c98c8fb04a91d1c2b0 in term 1.
I20260924 00:50:31.897177 8670 leader_election.cc:302] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df7ecfb4ee143c6b7f69865d39d14ef, 90f2d969822848c98c8fb04a91d1c2b0; no voters:
I20260924 00:50:31.897475 9140 raft_consensus.cc:2781] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:31.897851 9140 raft_consensus.cc:686] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 LEADER]: Becoming Leader. State: Replica: 90f2d969822848c98c8fb04a91d1c2b0, State: Running, Role: LEADER
I20260924 00:50:31.898115 9140 consensus_queue.cc:237] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } }
I20260924 00:50:31.900784 8579 catalog_manager.cc:5495] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1). New cstate: current_term: 1 leader_uuid: "90f2d969822848c98c8fb04a91d1c2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f2d969822848c98c8fb04a91d1c2b0" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 } health_report { overall_health: UNKNOWN } } }
I20260924 00:50:31.937288 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.932429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.0.222.1:33361)
0924 00:50:31.932866 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:31.932870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:31.932886 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:50:31.932941 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:31.932944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:31.933005 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:31.933085 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:31.933491 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.934222 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.935646 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.936056 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.936147 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.936560 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.936677 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.936743 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:31.936919 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 00:50:31.936984 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260924 00:50:31.951278 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.946222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:52721 (local address 127.0.222.1:33361)
0924 00:50:31.946370 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:31.946374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:31.946888 (+ 514us) server_negotiation.cc:407] Connection header received
0924 00:50:31.947028 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:31.947032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:31.947094 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:31.947166 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:50:31.947718 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.948442 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.949811 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.950223 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.950322 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.950638 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.950749 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.950819 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:31.950929 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 00:50:31.951020 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":43}
I20260924 00:50:31.951387 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:31.946115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33361 (local address 127.0.222.1:52721)
0924 00:50:31.946795 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:31.946812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:31.946935 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:31.947193 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:31.947196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:31.947218 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:50:31.947440 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:31.947447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.948597 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.948601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:31.949681 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:31.949693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:31.950371 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:31.950375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:31.950485 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:31.950522 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:50:31.950830 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:50:31.950838 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 00:50:31.950896 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 00:50:31.951016 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":3,"thread_start_us":400,"threads_started":1}
I20260924 00:50:31.959275 9025 raft_consensus.cc:1261] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Refusing update from remote peer 90f2d969822848c98c8fb04a91d1c2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:31.959362 8879 raft_consensus.cc:1261] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Refusing update from remote peer 90f2d969822848c98c8fb04a91d1c2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:50:31.959883 9184 consensus_queue.cc:1035] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df7ecfb4ee143c6b7f69865d39d14ef" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:31.960258 9182 consensus_queue.cc:1035] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bec43e620c64ca0aba75da41c21b190" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:50:32.012574 8581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:32.012809 8581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:50:32.016080 8581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e936dc660af14ad882b69ef2fb28b805: Sending DeleteTablet for 3 replicas of tablet f66796a34c3c4f94890daa8260da8c75
I20260924 00:50:32.016973 8720 tablet_service.cc:1514] Processing DeleteTablet for tablet f66796a34c3c4f94890daa8260da8c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:32 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49152
I20260924 00:50:32.017071 9001 tablet_service.cc:1514] Processing DeleteTablet for tablet f66796a34c3c4f94890daa8260da8c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:32 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:50188
I20260924 00:50:32.017359 9159 tablet_replica.cc:330] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: stopping tablet replica
I20260924 00:50:32.017407 9158 tablet_replica.cc:330] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: stopping tablet replica
I20260924 00:50:32.017581 9158 raft_consensus.cc:2229] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:32.017587 9159 raft_consensus.cc:2229] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:50:32.017788 9158 raft_consensus.cc:2258] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:32.018133 9159 raft_consensus.cc:2258] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:32.018759 8859 tablet_service.cc:1514] Processing DeleteTablet for tablet f66796a34c3c4f94890daa8260da8c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:50:32 UTC) from {username='oryx', principal='oryx/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:55334
I20260924 00:50:32.019068 9157 tablet_replica.cc:330] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: stopping tablet replica
I20260924 00:50:32.019124 9158 ts_tablet_manager.cc:1905] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:32.019245 9157 raft_consensus.cc:2229] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:50:32.019454 9157 raft_consensus.cc:2258] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:50:32.019884 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 8653
I20260924 00:50:32.020540 9159 ts_tablet_manager.cc:1905] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:32.025785 9158 ts_tablet_manager.cc:1918] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:50:32.026000 9157 ts_tablet_manager.cc:1905] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:50:32.026038 9158 log.cc:1196] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/f66796a34c3c4f94890daa8260da8c75
I20260924 00:50:32.026508 9158 ts_tablet_manager.cc:1939] T f66796a34c3c4f94890daa8260da8c75 P 1bec43e620c64ca0aba75da41c21b190: Deleting consensus metadata
I20260924 00:50:32.027761 8566 catalog_manager.cc:4841] TS 1bec43e620c64ca0aba75da41c21b190 (127.0.222.3:37627): tablet f66796a34c3c4f94890daa8260da8c75 (table test-table [id=86de7ad0cb76472a901575b45b83c464]) successfully deleted
I20260924 00:50:32.030714 9159 ts_tablet_manager.cc:1918] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:50:32.030948 9159 log.cc:1196] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/f66796a34c3c4f94890daa8260da8c75
I20260924 00:50:32.031462 9159 ts_tablet_manager.cc:1939] T f66796a34c3c4f94890daa8260da8c75 P 90f2d969822848c98c8fb04a91d1c2b0: Deleting consensus metadata
I20260924 00:50:32.032586 9157 ts_tablet_manager.cc:1918] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:50:32.032661 8565 catalog_manager.cc:4841] TS 90f2d969822848c98c8fb04a91d1c2b0 (127.0.222.1:33361): tablet f66796a34c3c4f94890daa8260da8c75 (table test-table [id=86de7ad0cb76472a901575b45b83c464]) successfully deleted
I20260924 00:50:32.032732 9157 log.cc:1196] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/f66796a34c3c4f94890daa8260da8c75
I20260924 00:50:32.033165 9157 ts_tablet_manager.cc:1939] T f66796a34c3c4f94890daa8260da8c75 P 2df7ecfb4ee143c6b7f69865d39d14ef: Deleting consensus metadata
I20260924 00:50:32.034379 8566 catalog_manager.cc:4841] TS 2df7ecfb4ee143c6b7f69865d39d14ef (127.0.222.2:33283): tablet f66796a34c3c4f94890daa8260da8c75 (table test-table [id=86de7ad0cb76472a901575b45b83c464]) successfully deleted
Sep 24 00:50:32 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.1@KRBTEST.COM
I20260924 00:50:32.047854 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:32.020857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.0.222.1:33361)
0924 00:50:32.021042 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:32.021049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:32.032955 (+ 11906us) server_negotiation.cc:407] Connection header received
0924 00:50:32.033301 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:32.033307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:32.033426 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:32.033499 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:32.035143 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:32.035928 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:32.037138 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:32.037635 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:32.037745 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:32.041726 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:32.041742 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:32.041758 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:32.041797 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:32.044904 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:32.045661 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:32.045671 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:32.045673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:32.045821 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:32.046202 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:32.046206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:32.046208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:32.046609 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:32.046930 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:32.047251 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 00:50:32.047510 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:50:32.048550 8785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:32.179644 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 8653
I20260924 00:50:32.199075 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 8794
Sep 24 00:50:32 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.2@KRBTEST.COM
I20260924 00:50:32.215432 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:32.199903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.0.222.2:33283)
0924 00:50:32.200383 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:32.200387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:32.200403 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:50:32.200459 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:32.200464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:32.200528 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:32.200640 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:32.201996 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:32.203130 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:32.204377 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:32.205025 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:32.205139 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:32.209475 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:32.209489 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:32.209506 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:32.209546 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:32.212359 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:32.213350 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:32.213370 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:32.213373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:32.213517 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:32.213944 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:32.213949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:32.213951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:32.214349 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:32.214587 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:32.214921 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 00:50:32.215097 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":234,"threads_started":1}
I20260924 00:50:32.216348 8925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:32.312259 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 8794
I20260924 00:50:32.327255 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 8934
Sep 24 00:50:32 dist-test-slave-rtvz krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.222.3@KRBTEST.COM
I20260924 00:50:32.344257 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:32.328228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.0.222.3:37627)
0924 00:50:32.328711 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:32.328716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:32.328731 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:50:32.328857 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:32.328869 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:32.328932 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:32.329029 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:32.330529 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:32.331317 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:32.332707 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:32.333147 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:32.333479 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:32.338178 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:32.338195 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:32.338213 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:32.338255 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:32.341130 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:32.342046 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:32.342054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:32.342056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:32.342213 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:32.342777 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:32.342781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:32.342783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:32.343175 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:32.343405 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:32.343739 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 00:50:32.344001 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":242,"threads_started":1}
I20260924 00:50:32.345064 9067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:32.470057 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 8934
I20260924 00:50:32.487752 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 8549
I20260924 00:50:32.489145 8622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:32.606163 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 8549
2026-09-24T00:50:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5820 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:50:32 dist-test-slave-rtvz krb5kdc[9203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:50:32 dist-test-slave-rtvz krb5kdc[9203](info): set up 2 sockets
Sep 24 00:50:32 dist-test-slave-rtvz krb5kdc[9203](info): commencing operation
krb5kdc: starting...
W20260924 00:50:33.621243 888 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:50:33 dist-test-slave-rtvz krb5kdc[9203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:50:33.836998 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41207
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:34.167027 9216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:34.167326 9216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:34.167416 9216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:34.174921 9216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:34.175045 9216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:34.175091 9216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:34.175144 9216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:34.175190 9216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:34.185015 9216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:34.186290 9216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:34.187585 9216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:34.199499 9221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:34.199729 9222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:34.199733 9224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:34.200825 9216 server_base.cc:1034] running on GCE node
I20260924 00:50:34.201772 9216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:34.203250 9216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:34.204502 9216 hybrid_clock.cc:648] HybridClock initialized: now 1790211034204435 us; error 42 us; skew 500 ppm
I20260924 00:50:34.204878 9216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T00:50:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1691 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:50:34 dist-test-slave-rtvz krb5kdc[9230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:50:34 dist-test-slave-rtvz krb5kdc[9230](info): set up 2 sockets
Sep 24 00:50:34 dist-test-slave-rtvz krb5kdc[9230](info): commencing operation
krb5kdc: starting...
W20260924 00:50:35.317099 888 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:50:35 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:50:35.536809 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42561
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:35.872427 9243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:35.872792 9243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:35.872893 9243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:35.880968 9243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:35.881114 9243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:35.881163 9243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:35.881254 9243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:35.881309 9243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:35.891664 9243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:35.893010 9243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:35.894322 9243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:35.905905 9248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:35.905954 9249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:35.906103 9251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:35.906968 9243 server_base.cc:1034] running on GCE node
I20260924 00:50:35.907770 9243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:35.909332 9243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:35.910558 9243 hybrid_clock.cc:648] HybridClock initialized: now 1790211035910510 us; error 35 us; skew 500 ppm
I20260924 00:50:35.910928 9243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:35 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:35.914371 9243 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:50:35.916141 9243 webserver.cc:460] Webserver started at http://127.0.222.62:33587/ using document root <none> and password file <none>
I20260924 00:50:35.916888 9243 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:35.917029 9243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:35.917392 9243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:35.920645 9243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "666568f1fbf645789a887b1f8d7728d5"
format_stamp: "Formatted at 2026-09-24 00:50:35 on dist-test-slave-rtvz"
server_key: "67c694725dcf8d5aa8cda7a3a88d4334"
server_key_iv: "6138e93557c42e482f206106f18d3630"
server_key_version: "encryptionkey@0"
I20260924 00:50:35.921517 9243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "666568f1fbf645789a887b1f8d7728d5"
format_stamp: "Formatted at 2026-09-24 00:50:35 on dist-test-slave-rtvz"
server_key: "67c694725dcf8d5aa8cda7a3a88d4334"
server_key_iv: "6138e93557c42e482f206106f18d3630"
server_key_version: "encryptionkey@0"
I20260924 00:50:35.927203 9243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:50:35.931106 9258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:35.932564 9243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:35.932798 9243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "666568f1fbf645789a887b1f8d7728d5"
format_stamp: "Formatted at 2026-09-24 00:50:35 on dist-test-slave-rtvz"
server_key: "67c694725dcf8d5aa8cda7a3a88d4334"
server_key_iv: "6138e93557c42e482f206106f18d3630"
server_key_version: "encryptionkey@0"
I20260924 00:50:35.932996 9243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:35.950536 9243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:35.954939 9243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:35.955235 9243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:35.973656 9243 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42561
I20260924 00:50:35.973685 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42561 every 8 connection(s)
I20260924 00:50:35.975020 9243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:35.978478 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 9243
I20260924 00:50:35.978694 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:35.979236 9312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:35.979490 888 external_mini_cluster.cc:1428] Setting key 4decbe5877e5a77082e78d8982a7691e
I20260924 00:50:35.994351 9312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Bootstrap starting.
Sep 24 00:50:35 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:35.998312 9312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:35.999712 9312 log.cc:824] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:36.003305 9312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: No bootstrap required, opened a new log
I20260924 00:50:36.005067 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:35.981674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.0.222.62:42561)
0924 00:50:35.982546 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:35.982558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:35.982597 (+ 39us) server_negotiation.cc:407] Connection header received
0924 00:50:35.984336 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:35.984359 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:35.984779 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:35.985122 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:35.986948 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:35.988375 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:35.990180 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:35.991054 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:35.991211 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:35.995925 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:35.995941 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:35.995965 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:35.996018 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:36.000018 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:36.001081 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:36.001088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:36.001094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:36.001345 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:36.001814 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:36.001819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:36.001821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:36.002533 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:36.002870 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:36.003251 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 00:50:36.003696 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":415,"threads_started":1}
I20260924 00:50:36.011338 9312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } }
I20260924 00:50:36.011698 9312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:36.011770 9312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666568f1fbf645789a887b1f8d7728d5, State: Initialized, Role: FOLLOWER
I20260924 00:50:36.012252 9312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } }
I20260924 00:50:36.012456 9312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:36.012553 9312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:36.012671 9312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:36.014891 9312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } }
I20260924 00:50:36.015362 9312 leader_election.cc:302] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666568f1fbf645789a887b1f8d7728d5; no voters:
I20260924 00:50:36.015926 9312 leader_election.cc:288] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:36.016072 9317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:36.016686 9317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 1 LEADER]: Becoming Leader. State: Replica: 666568f1fbf645789a887b1f8d7728d5, State: Running, Role: LEADER
I20260924 00:50:36.017411 9317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } }
I20260924 00:50:36.018196 9312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:36.019979 9318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "666568f1fbf645789a887b1f8d7728d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } } }
I20260924 00:50:36.019951 9319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666568f1fbf645789a887b1f8d7728d5. Latest consensus state: current_term: 1 leader_uuid: "666568f1fbf645789a887b1f8d7728d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } } }
I20260924 00:50:36.020411 9318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:36.020526 9319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:36.022639 9322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:36.028175 9322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:36.037807 9322 catalog_manager.cc:1347] Generated new cluster ID: ceb02405d9604afda68ffb5adfc91ba3
I20260924 00:50:36.037968 9322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:36.072105 9322 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:36.073639 9322 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:36.088716 9322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Generated new TSK 0
I20260924 00:50:36.090145 9322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:50:36.151029 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42561
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:36.475975 9338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:36.476257 9338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:36.476364 9338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:36.483742 9338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:36.483857 9338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:36.483970 9338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:36.493813 9338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:36.495051 9338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:36.496330 9338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:36.509523 9344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:36.509543 9343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:36.509570 9346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:36.510294 9338 server_base.cc:1034] running on GCE node
I20260924 00:50:36.510931 9338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:36.511912 9338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:36.513108 9338 hybrid_clock.cc:648] HybridClock initialized: now 1790211036513053 us; error 47 us; skew 500 ppm
I20260924 00:50:36.513459 9338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:36 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:36.516552 9338 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:50:36.518049 9338 webserver.cc:460] Webserver started at http://127.0.222.1:35651/ using document root <none> and password file <none>
I20260924 00:50:36.518767 9338 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:36.518885 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:36.519210 9338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:36.522187 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "bb080294f6aa497e8036bb8817351fc7"
format_stamp: "Formatted at 2026-09-24 00:50:36 on dist-test-slave-rtvz"
server_key: "de79dce28086adb51d63d07d3c82884d"
server_key_iv: "78b679e99371d25a587648320a8453d2"
server_key_version: "encryptionkey@0"
I20260924 00:50:36.522948 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "bb080294f6aa497e8036bb8817351fc7"
format_stamp: "Formatted at 2026-09-24 00:50:36 on dist-test-slave-rtvz"
server_key: "de79dce28086adb51d63d07d3c82884d"
server_key_iv: "78b679e99371d25a587648320a8453d2"
server_key_version: "encryptionkey@0"
I20260924 00:50:36.528134 9338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:50:36.531776 9353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:36.533265 9338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:36.533444 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "bb080294f6aa497e8036bb8817351fc7"
format_stamp: "Formatted at 2026-09-24 00:50:36 on dist-test-slave-rtvz"
server_key: "de79dce28086adb51d63d07d3c82884d"
server_key_iv: "78b679e99371d25a587648320a8453d2"
server_key_version: "encryptionkey@0"
I20260924 00:50:36.533635 9338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:36.552690 9338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:36.557138 9338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:36.557457 9338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:36.558450 9338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:36.559828 9338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:36.559919 9338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:36.560074 9338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:36.560134 9338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:36.590516 9338 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:40125
I20260924 00:50:36.590544 9467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:40125 every 8 connection(s)
I20260924 00:50:36.591881 9338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:36.601742 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 9338
I20260924 00:50:36.601946 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:50:36.602766 888 external_mini_cluster.cc:1428] Setting key f453f6c8aaac879f3749fa5716a8a267
Sep 24 00:50:36 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:36.611851 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:36.594349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:57575 (local address 127.0.222.62:42561)
0924 00:50:36.594794 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:36.594798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:36.595739 (+ 941us) server_negotiation.cc:407] Connection header received
0924 00:50:36.597479 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:36.597485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:36.597573 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:36.597678 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:36.599162 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:36.599865 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:36.601466 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:36.601918 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:36.602043 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:36.605854 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:36.605860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:36.605866 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:36.605905 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:36.608839 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:36.609587 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:36.609591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:36.609594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:36.609748 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:36.610161 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:36.610165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:36.610168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:36.610478 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:36.610610 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:36.611282 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 00:50:36.611498 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":194,"threads_started":1}
I20260924 00:50:36.612912 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:36.594826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:57575)
0924 00:50:36.595536 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:36.595578 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:36.597230 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:36.597907 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:36.597915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:36.598434 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:36.598976 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:36.599015 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:36.600003 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:36.600007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:36.601324 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:36.601340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:36.602178 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:36.602187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:36.602479 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:36.603175 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:36.603210 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:36.605615 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:36.608974 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:36.608981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:36.608995 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:36.609476 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:36.609869 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:36.609872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:36.609875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:36.610058 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:36.610595 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:36.610602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:36.610935 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 00:50:36.611156 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 00:50:36.611545 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":286,"threads_started":1}
I20260924 00:50:36.614569 9468 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42561
I20260924 00:50:36.614882 9468 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:36.615605 9468 heartbeater.cc:511] Master 127.0.222.62:42561 requested a full tablet report, sending...
I20260924 00:50:36.617722 9275 ts_manager.cc:194] Registered new tserver with Master: bb080294f6aa497e8036bb8817351fc7 (127.0.222.1:40125)
I20260924 00:50:36.619206 9275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:57575
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:50:36.650771 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42561
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:36.976845 9474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:36.977131 9474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:36.977284 9474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:36.984871 9474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:36.985002 9474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:36.985162 9474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:36.995505 9474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:36.996903 9474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:36.998394 9474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:37.011768 9479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:37.011790 9480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:37.011924 9482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:37.012832 9474 server_base.cc:1034] running on GCE node
I20260924 00:50:37.013532 9474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:37.014511 9474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:37.015724 9474 hybrid_clock.cc:648] HybridClock initialized: now 1790211037015664 us; error 44 us; skew 500 ppm
I20260924 00:50:37.016031 9474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:37 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:37.019359 9474 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:50:37.020905 9474 webserver.cc:460] Webserver started at http://127.0.222.2:40329/ using document root <none> and password file <none>
I20260924 00:50:37.021627 9474 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:37.021732 9474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:37.022006 9474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:37.024993 9474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "97222b245ce64aacbc5cafb71fc200a1"
format_stamp: "Formatted at 2026-09-24 00:50:37 on dist-test-slave-rtvz"
server_key: "c7413e74d8bcdd49a7b7b71b9cd8e939"
server_key_iv: "2f00af02ae050d71508153f93b8a1626"
server_key_version: "encryptionkey@0"
I20260924 00:50:37.025838 9474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "97222b245ce64aacbc5cafb71fc200a1"
format_stamp: "Formatted at 2026-09-24 00:50:37 on dist-test-slave-rtvz"
server_key: "c7413e74d8bcdd49a7b7b71b9cd8e939"
server_key_iv: "2f00af02ae050d71508153f93b8a1626"
server_key_version: "encryptionkey@0"
I20260924 00:50:37.031560 9474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 00:50:37.035333 9489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:37.036868 9474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:37.037036 9474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "97222b245ce64aacbc5cafb71fc200a1"
format_stamp: "Formatted at 2026-09-24 00:50:37 on dist-test-slave-rtvz"
server_key: "c7413e74d8bcdd49a7b7b71b9cd8e939"
server_key_iv: "2f00af02ae050d71508153f93b8a1626"
server_key_version: "encryptionkey@0"
I20260924 00:50:37.037307 9474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:37.057802 9474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:37.062638 9474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:37.062943 9474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:37.063962 9474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:37.065501 9474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:37.065585 9474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:37.065718 9474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:37.065771 9474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:37.097924 9474 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:45491
I20260924 00:50:37.097986 9602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:45491 every 8 connection(s)
I20260924 00:50:37.099224 9474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:37.101526 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 9474
I20260924 00:50:37.101715 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:50:37.102263 888 external_mini_cluster.cc:1428] Setting key ed6b145ef296f7638d9d9d31b6f2c313
Sep 24 00:50:37 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:37.120936 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:37.102081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:57023 (local address 127.0.222.62:42561)
0924 00:50:37.102311 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:37.102317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:37.103635 (+ 1318us) server_negotiation.cc:407] Connection header received
0924 00:50:37.105741 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:37.105747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:37.105830 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:37.105927 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:37.107546 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.108477 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.110105 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.110736 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.110869 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:37.114987 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:37.114992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:37.114995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:37.115030 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:37.117591 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:37.118381 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:37.118385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:37.118388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:37.118513 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:37.118943 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:37.118947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:37.118949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:37.119254 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:37.119400 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:37.120336 (+ 936us) server_negotiation.cc:299] Negotiation successful
0924 00:50:37.120590 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:50:37.122022 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:37.102621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:57023)
0924 00:50:37.103462 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:37.103512 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:37.105477 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:37.106199 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:37.106208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:37.106797 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:37.107386 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:37.107403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.108639 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.108644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:37.109968 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:37.109981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.111002 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.111009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:37.111333 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:37.112128 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:37.112165 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:37.114747 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:37.117740 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:37.117761 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:37.117780 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:37.118239 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:37.118615 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:37.118620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:37.118624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:37.118826 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:37.119446 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:37.119455 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:37.119935 (+ 480us) client_negotiation.cc:769] Sending connection context
0924 00:50:37.120239 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 00:50:37.120813 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":314,"threads_started":1}
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:50:37.123553 9603 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42561
I20260924 00:50:37.123966 9603 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:37.124689 9603 heartbeater.cc:511] Master 127.0.222.62:42561 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:50:37.126279 9275 ts_manager.cc:194] Registered new tserver with Master: 97222b245ce64aacbc5cafb71fc200a1 (127.0.222.2:45491)
I20260924 00:50:37.127094 9275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:57023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:50:37.152495 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42561
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:37.478636 9608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:37.478953 9608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:37.479079 9608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:37.486589 9608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:37.486728 9608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:37.486882 9608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:37.496982 9608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:37.498346 9608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:37.499653 9608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:37.513625 9613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:37.513587 9616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:37.513656 9614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:37.514201 9608 server_base.cc:1034] running on GCE node
I20260924 00:50:37.514901 9608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:37.515915 9608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:37.517153 9608 hybrid_clock.cc:648] HybridClock initialized: now 1790211037517091 us; error 46 us; skew 500 ppm
I20260924 00:50:37.517513 9608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:37 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:37.520789 9608 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:50:37.522369 9608 webserver.cc:460] Webserver started at http://127.0.222.3:38567/ using document root <none> and password file <none>
I20260924 00:50:37.523123 9608 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:37.523253 9608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:37.523554 9608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:37.526733 9608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "1e4335e8db3e4610a012cbd853c5d37a"
format_stamp: "Formatted at 2026-09-24 00:50:37 on dist-test-slave-rtvz"
server_key: "7db9818f633e8025ce3bed6f50bb42e6"
server_key_iv: "e600db76329c9a646ff796fbda6bcffd"
server_key_version: "encryptionkey@0"
I20260924 00:50:37.527573 9608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4335e8db3e4610a012cbd853c5d37a"
format_stamp: "Formatted at 2026-09-24 00:50:37 on dist-test-slave-rtvz"
server_key: "7db9818f633e8025ce3bed6f50bb42e6"
server_key_iv: "e600db76329c9a646ff796fbda6bcffd"
server_key_version: "encryptionkey@0"
I20260924 00:50:37.533087 9608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 00:50:37.537168 9623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:37.538947 9608 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 00:50:37.539155 9608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "1e4335e8db3e4610a012cbd853c5d37a"
format_stamp: "Formatted at 2026-09-24 00:50:37 on dist-test-slave-rtvz"
server_key: "7db9818f633e8025ce3bed6f50bb42e6"
server_key_iv: "e600db76329c9a646ff796fbda6bcffd"
server_key_version: "encryptionkey@0"
I20260924 00:50:37.539417 9608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:37.557986 9608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:37.561947 9608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:37.562212 9608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:37.563222 9608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:37.564534 9608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:37.564606 9608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:37.564698 9608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:37.564741 9608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:37.597528 9608 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:45227
I20260924 00:50:37.597568 9735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:45227 every 8 connection(s)
I20260924 00:50:37.598855 9608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:37.603507 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 9608
I20260924 00:50:37.603709 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:50:37.604233 888 external_mini_cluster.cc:1428] Setting key 5793aba54914aa0fe411c7457a9168cc
Sep 24 00:50:37 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:50:37.620368 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:37.601386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:58945 (local address 127.0.222.62:42561)
0924 00:50:37.601549 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:37.601554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:37.602732 (+ 1178us) server_negotiation.cc:407] Connection header received
0924 00:50:37.605151 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:37.605156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:37.605306 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:37.605411 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:37.607451 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.608315 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.609939 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.610339 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.610447 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:37.613896 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:37.613901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:37.613905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:37.613940 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:37.616587 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:37.617499 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:37.617503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:37.617505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:37.617647 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:37.618139 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:37.618143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:37.618146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:37.618508 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:37.618655 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:37.619717 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0924 00:50:37.619963 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 00:50:37.621506 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:37.601846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:58945)
0924 00:50:37.602561 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:37.602599 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:37.604854 (+ 2255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:37.605706 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:37.605720 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:37.606523 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:37.607258 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:37.607285 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.608476 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.608482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:37.609805 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:37.609817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.610566 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.610574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:37.610819 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:37.611446 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:37.611474 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:37.613669 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:37.616781 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:37.616789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:37.616805 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:37.617354 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:37.617791 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:50:37.617794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:50:37.617797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:50:37.618012 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:50:37.618657 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:37.618666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:37.619213 (+ 547us) client_negotiation.cc:769] Sending connection context
0924 00:50:37.619576 (+ 363us) client_negotiation.cc:240] Negotiation successful
0924 00:50:37.620129 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":283,"threads_started":1}
I20260924 00:50:37.622357 9468 heartbeater.cc:503] Master 127.0.222.62:42561 was elected leader, sending a full tablet report...
I20260924 00:50:37.623243 9736 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42561
I20260924 00:50:37.623570 9736 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:37.624392 9736 heartbeater.cc:511] Master 127.0.222.62:42561 requested a full tablet report, sending...
I20260924 00:50:37.625941 9275 ts_manager.cc:194] Registered new tserver with Master: 1e4335e8db3e4610a012cbd853c5d37a (127.0.222.3:45227)
I20260924 00:50:37.626842 9275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:58945
I20260924 00:50:37.633417 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab.
Entry for principal oryx/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab.
WARNING: no policy specified for oryx/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab.
Entry for principal oryx/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab.
WARNING: no policy specified for oryx/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab.
Entry for principal oryx/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab.
I20260924 00:50:37.776609 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9338
Sep 24 00:50:37 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:50:37.793704 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:37.777789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.0.222.1:40125)
0924 00:50:37.778405 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:37.778412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:37.778475 (+ 63us) server_negotiation.cc:407] Connection header received
0924 00:50:37.778540 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:37.778546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:37.778763 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:37.778892 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:37.780311 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.781468 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.782769 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.783358 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.783467 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:37.787661 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:37.787678 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:37.787695 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:37.787742 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:37.790655 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:37.791506 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:37.791514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:37.791517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:37.791767 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:37.792217 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:37.792221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:37.792223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:37.792603 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:37.792850 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:37.793116 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 00:50:37.793405 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":294,"threads_started":1}
I20260924 00:50:37.795125 9463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:37.878099 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 9338
I20260924 00:50:37.891552 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9474
Sep 24 00:50:37 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:50:37.908552 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:37.892553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.0.222.2:45491)
0924 00:50:37.893099 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:37.893105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:37.893132 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:50:37.893231 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:37.893241 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:37.893475 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:37.893610 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:37.894917 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.895958 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.897411 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:37.898015 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:37.898122 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:37.902598 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:37.902613 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:37.902629 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:37.902678 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:37.905457 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:37.906267 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:37.906275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:37.906277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:37.906425 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:37.906900 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:37.906903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:37.906906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:37.907312 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:37.907560 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:37.907939 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 00:50:37.908197 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":280,"threads_started":1}
I20260924 00:50:37.909855 9597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:37.998680 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 9474
I20260924 00:50:38.011651 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9608
Sep 24 00:50:38 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:50:38.029922 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:38.012702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.0.222.3:45227)
0924 00:50:38.013352 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:38.013360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:38.013389 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:50:38.013448 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:38.013455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:38.013664 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:38.013814 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:38.015199 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:38.016325 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:38.017821 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:38.018421 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:38.018540 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:38.023389 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:38.023443 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:38.023465 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:38.023516 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:38.026803 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:38.027657 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:38.027666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:38.027668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:38.027820 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:38.028224 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:38.028229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:38.028231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:38.028626 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:38.028893 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:38.029252 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 00:50:38.029483 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":284,"threads_started":1}
I20260924 00:50:38.031322 9731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:38.114288 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 9608
I20260924 00:50:38.127130 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9243
I20260924 00:50:38.128429 9306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:38.233574 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 9243
I20260924 00:50:38.250455 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42561
--webserver_interface=127.0.222.62
--webserver_port=33587
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:50:38.584709 9754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:38.585143 9754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:38.585347 9754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:38.594624 9754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:38.594841 9754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:38.594944 9754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:38.595033 9754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:38.595108 9754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:38.607749 9754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=33587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:38.609504 9754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:38.611119 9754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:38.624806 9762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:38.624907 9759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:38.624943 9760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:38.625465 9754 server_base.cc:1034] running on GCE node
I20260924 00:50:38.626313 9754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:38.627761 9754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:38.629001 9754 hybrid_clock.cc:648] HybridClock initialized: now 1790211038628938 us; error 47 us; skew 500 ppm
I20260924 00:50:38.629421 9754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:38 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:38.632431 9754 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:50:38.633868 9754 webserver.cc:460] Webserver started at http://127.0.222.62:33587/ using document root <none> and password file <none>
I20260924 00:50:38.634514 9754 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:38.634625 9754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:38.640056 9754 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 00:50:38.642944 9769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:38.644397 9754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:50:38.644567 9754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "666568f1fbf645789a887b1f8d7728d5"
format_stamp: "Formatted at 2026-09-24 00:50:35 on dist-test-slave-rtvz"
server_key: "67c694725dcf8d5aa8cda7a3a88d4334"
server_key_iv: "6138e93557c42e482f206106f18d3630"
server_key_version: "encryptionkey@0"
I20260924 00:50:38.645639 9754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:38.658766 9754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:38.662891 9754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:38.663141 9754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:38.680980 9754 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42561
I20260924 00:50:38.680968 9820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42561 every 8 connection(s)
I20260924 00:50:38.682667 9754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:38.689682 9821 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:50:38.691484 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 9754
I20260924 00:50:38.692216 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:40125
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:42561
--webserver_port=35651
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.222.1
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:50:38.693186 9821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Bootstrap starting.
I20260924 00:50:38.712711 9821 log.cc:824] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:38.721963 9821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:50:38.722474 9821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Bootstrap complete.
I20260924 00:50:38.731593 9821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } }
I20260924 00:50:38.732501 9821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666568f1fbf645789a887b1f8d7728d5, State: Initialized, Role: FOLLOWER
I20260924 00:50:38.733168 9821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } }
I20260924 00:50:38.733476 9821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:38.733589 9821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:38.733758 9821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:50:38.737768 9821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } }
I20260924 00:50:38.738332 9821 leader_election.cc:302] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666568f1fbf645789a887b1f8d7728d5; no voters:
I20260924 00:50:38.738986 9821 leader_election.cc:288] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:50:38.739151 9825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:50:38.740377 9825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [term 2 LEADER]: Becoming Leader. State: Replica: 666568f1fbf645789a887b1f8d7728d5, State: Running, Role: LEADER
I20260924 00:50:38.741150 9825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } }
I20260924 00:50:38.741459 9821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:38.743849 9827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666568f1fbf645789a887b1f8d7728d5. Latest consensus state: current_term: 2 leader_uuid: "666568f1fbf645789a887b1f8d7728d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } } }
I20260924 00:50:38.744323 9827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:38.744938 9826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "666568f1fbf645789a887b1f8d7728d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666568f1fbf645789a887b1f8d7728d5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42561 } } }
I20260924 00:50:38.745453 9826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:38.748957 9833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:38.754871 9833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:38.756583 9833 catalog_manager.cc:1259] Loaded cluster ID: ceb02405d9604afda68ffb5adfc91ba3
I20260924 00:50:38.756870 9833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:38.761595 9833 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:38.763588 9833 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 666568f1fbf645789a887b1f8d7728d5: Loaded TSK: 0
I20260924 00:50:38.765600 9833 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:50:39.074048 9823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:39.074364 9823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:39.074508 9823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:39.082168 9823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:39.082305 9823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:39.082473 9823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:39.092803 9823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.1.keytab
--principal=oryx/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=35651
--tserver_master_addrs=127.0.222.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:39.094266 9823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:39.095651 9823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:39.109884 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:39.109910 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:39.110402 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:39.110605 9823 server_base.cc:1034] running on GCE node
I20260924 00:50:39.111474 9823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:39.112475 9823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:39.113734 9823 hybrid_clock.cc:648] HybridClock initialized: now 1790211039113659 us; error 55 us; skew 500 ppm
I20260924 00:50:39.114073 9823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211039, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:39.117424 9823 init.cc:377] Logged in from keytab as oryx/127.0.222.1@KRBTEST.COM (short username oryx)
I20260924 00:50:39.118997 9823 webserver.cc:460] Webserver started at http://127.0.222.1:35651/ using document root <none> and password file <none>
I20260924 00:50:39.119731 9823 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:39.119871 9823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:39.125892 9823 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 00:50:39.128908 9857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:39.130555 9823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:39.130730 9823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "bb080294f6aa497e8036bb8817351fc7"
format_stamp: "Formatted at 2026-09-24 00:50:36 on dist-test-slave-rtvz"
server_key: "de79dce28086adb51d63d07d3c82884d"
server_key_iv: "78b679e99371d25a587648320a8453d2"
server_key_version: "encryptionkey@0"
I20260924 00:50:39.131737 9823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:39.145042 9823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:39.150352 9823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:39.150616 9823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:39.151598 9823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:39.152941 9823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:39.153033 9823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:39.153138 9823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:39.153231 9823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:39.182377 9823 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:40125
I20260924 00:50:39.182456 9971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:40125 every 8 connection(s)
I20260924 00:50:39.183880 9823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:39.184435 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 9823
I20260924 00:50:39.185385 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:45491
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:42561
--webserver_port=40329
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.222.2
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:50:39.207492 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:39.187195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:35027 (local address 127.0.222.62:42561)
0924 00:50:39.192462 (+ 5267us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:39.192475 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:39.192515 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:50:39.192847 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:39.192862 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:39.193296 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:39.193690 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:39.195636 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.196965 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.199321 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.200075 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.200233 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.206481 (+ 6248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:35027: BlockingRecv error: recv got EOF from 127.0.222.1:35027 (error 108)
Metrics: {"server-negotiator.queue_time_us":5020,"thread_start_us":4841,"threads_started":1}
I20260924 00:50:39.207549 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:39.187113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:35027)
0924 00:50:39.189855 (+ 2742us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:39.189911 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:39.191697 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:39.193880 (+ 2183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:39.193891 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:39.194643 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:39.195386 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.195415 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.197144 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.197149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.199125 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.199143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.200326 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.200333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.200596 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.201387 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:39.201416 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:39.206129 (+ 4713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2412,"thread_start_us":2201,"threads_started":1}
W20260924 00:50:39.208214 9972 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42561 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.222.62:42561: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:39.217623 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:39.208871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:52863)
0924 00:50:39.209078 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:39.209099 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:39.209251 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:39.209808 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:39.209812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:39.210146 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:39.210301 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.210309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.211775 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.211779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.212742 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.212755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.213773 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.213778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.213929 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.214563 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:39.214583 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:39.217398 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 00:50:39.218071 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:39.209169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:52863 (local address 127.0.222.62:42561)
0924 00:50:39.209390 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:39.209396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:39.209417 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:50:39.209482 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:39.209488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:39.209568 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:39.209674 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:39.210457 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.211616 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.212870 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.213553 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.213672 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.217668 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:52863: BlockingRecv error: recv got EOF from 127.0.222.1:52863 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:39.227360 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:39.218371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:42455)
0924 00:50:39.218582 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:39.218601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:39.218709 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:39.219023 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:39.219027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:39.219432 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:39.219624 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.219636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.220813 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.220819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.222198 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.222214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.223040 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.223046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.223206 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.223867 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:39.223889 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:39.227053 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 00:50:39.227703 9972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:50:39.227736 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:39.218485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:42455 (local address 127.0.222.62:42561)
0924 00:50:39.218653 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:39.218657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:39.218675 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:50:39.218816 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:39.218819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:39.218894 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:39.218973 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:39.219840 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.220582 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.222379 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.222886 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.222999 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.227326 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:42455: BlockingRecv error: recv got EOF from 127.0.222.1:42455 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 00:50:39.521284 9977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:39.521612 9977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:39.521760 9977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:39.529517 9977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:39.529665 9977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:39.529841 9977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:39.540408 9977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.2.keytab
--principal=oryx/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:45491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=40329
--tserver_master_addrs=127.0.222.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:39.541865 9977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:39.543171 9977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:39.556754 9982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:39.556901 9985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:39.556857 9983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:39.557804 9977 server_base.cc:1034] running on GCE node
I20260924 00:50:39.558485 9977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:39.559551 9977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:39.560776 9977 hybrid_clock.cc:648] HybridClock initialized: now 1790211039560707 us; error 66 us; skew 500 ppm
I20260924 00:50:39.561098 9977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211039, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:39.574324 9977 init.cc:377] Logged in from keytab as oryx/127.0.222.2@KRBTEST.COM (short username oryx)
I20260924 00:50:39.575975 9977 webserver.cc:460] Webserver started at http://127.0.222.2:40329/ using document root <none> and password file <none>
I20260924 00:50:39.576695 9977 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:39.576824 9977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:39.582832 9977 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 00:50:39.585777 9992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:39.587154 9977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:39.587333 9977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "97222b245ce64aacbc5cafb71fc200a1"
format_stamp: "Formatted at 2026-09-24 00:50:37 on dist-test-slave-rtvz"
server_key: "c7413e74d8bcdd49a7b7b71b9cd8e939"
server_key_iv: "2f00af02ae050d71508153f93b8a1626"
server_key_version: "encryptionkey@0"
I20260924 00:50:39.588294 9977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:39.604744 9977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:39.608949 9977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:39.609232 9977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:39.610227 9977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:39.611565 9977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:39.611649 9977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:39.611748 9977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:39.611816 9977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:39.642339 9977 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:45491
I20260924 00:50:39.642406 10106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:45491 every 8 connection(s)
I20260924 00:50:39.643847 9977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:39.651494 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 9977
I20260924 00:50:39.652153 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:45227
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:42561
--webserver_port=38567
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.222.3
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:50:39.659145 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:39.646425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:54179 (local address 127.0.222.62:42561)
0924 00:50:39.646596 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:39.646601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:39.647740 (+ 1139us) server_negotiation.cc:407] Connection header received
0924 00:50:39.649679 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:39.649685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:39.649773 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:39.649866 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:39.651428 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.652157 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.653783 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.654194 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.654306 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.658903 (+ 4597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:54179: BlockingRecv error: recv got EOF from 127.0.222.2:54179 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:50:39.659850 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:39.646857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:54179)
0924 00:50:39.647560 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:39.647617 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:39.649404 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:39.650116 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:39.650124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:39.650675 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:39.651247 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.651264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.652319 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.652324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.653635 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.653647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.654413 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.654420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.654649 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.655258 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:39.655288 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:39.658683 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":271,"threads_started":1}
W20260924 00:50:39.660501 10107 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42561 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.222.62:42561: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:39.668490 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:39.661064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:40193)
0924 00:50:39.661276 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:39.661295 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:39.661397 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:39.661689 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:39.661693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:39.662052 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:39.662191 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.662200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.663128 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.663132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.664006 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.664017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.664694 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.664700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.664867 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.665460 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:39.665477 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:39.668209 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 00:50:39.668663 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:39.661154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:40193 (local address 127.0.222.62:42561)
0924 00:50:39.661366 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:39.661371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:39.661388 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:50:39.661510 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:39.661515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:39.661577 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:39.661656 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:39.662331 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.663004 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.664131 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.664556 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.664670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.668446 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:40193: BlockingRecv error: recv got EOF from 127.0.222.2:40193 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:39 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:39.676702 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:39.669457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:54861)
0924 00:50:39.669633 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:39.669652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:39.669767 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:39.670103 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:39.670108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:39.670405 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:39.670551 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.670560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.671526 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.671531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.672434 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:39.672445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.673089 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.673095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:39.673313 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.673901 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:39.673916 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:39.676416 (+ 2500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 00:50:39.676841 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:39.669520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:54861 (local address 127.0.222.62:42561)
0924 00:50:39.669667 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:39.669672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:39.669706 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:50:39.669918 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:39.669924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:39.669989 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:39.670065 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:39.670683 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.671395 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.672561 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:39.672962 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:39.673067 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:39.676637 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:54861: BlockingRecv error: recv got EOF from 127.0.222.2:54861 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 00:50:39.677140 10107 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:50:39.984187 10110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:39.984489 10110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:39.984583 10110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:39.991998 10110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:39.992116 10110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:39.992242 10110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:40.002321 10110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/krb5kdc/oryx_127.0.222.3.keytab
--principal=oryx/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:45227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=38567
--tserver_master_addrs=127.0.222.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:40.003650 10110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:40.004988 10110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:40.018707 10116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:40.018675 10115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:40.018788 10118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:40.019661 10110 server_base.cc:1034] running on GCE node
I20260924 00:50:40.020324 10110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:40.021441 10110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:40.022709 10110 hybrid_clock.cc:648] HybridClock initialized: now 1790211040022634 us; error 57 us; skew 500 ppm
I20260924 00:50:40.023087 10110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211040, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:50:40.026458 10110 init.cc:377] Logged in from keytab as oryx/127.0.222.3@KRBTEST.COM (short username oryx)
I20260924 00:50:40.028223 10110 webserver.cc:460] Webserver started at http://127.0.222.3:38567/ using document root <none> and password file <none>
I20260924 00:50:40.028950 10110 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:40.029086 10110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:40.034762 10110 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 00:50:40.037748 10125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:40.039243 10110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:50:40.039412 10110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "1e4335e8db3e4610a012cbd853c5d37a"
format_stamp: "Formatted at 2026-09-24 00:50:37 on dist-test-slave-rtvz"
server_key: "7db9818f633e8025ce3bed6f50bb42e6"
server_key_iv: "e600db76329c9a646ff796fbda6bcffd"
server_key_version: "encryptionkey@0"
I20260924 00:50:40.040445 10110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:40.074836 10110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:40.079301 10110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:40.079566 10110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:40.080629 10110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:40.082032 10110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:40.082136 10110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:40.082250 10110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:40.082319 10110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:40.114013 10110 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:45227
I20260924 00:50:40.114056 10239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:45227 every 8 connection(s)
I20260924 00:50:40.115603 10110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:40.124147 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 10110
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:50:40.136145 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:40.118990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:56771 (local address 127.0.222.62:42561)
0924 00:50:40.119203 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:40.119209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:40.120586 (+ 1377us) server_negotiation.cc:407] Connection header received
0924 00:50:40.122745 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:40.122751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:40.122869 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:40.123120 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:40.125140 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.126434 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.128395 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.129026 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.129158 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.135851 (+ 6693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:56771: BlockingRecv error: recv got EOF from 127.0.222.3:56771 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 00:50:40.137517 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:40.119521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:56771)
0924 00:50:40.120399 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:40.120451 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:40.122466 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:40.123316 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:40.123330 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:40.124109 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:40.124940 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.124967 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.126609 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.126615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.128258 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.128270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.129677 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.129685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.129959 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.130710 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:40.130742 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:40.135620 (+ 4878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":317,"threads_started":1}
W20260924 00:50:40.138188 10240 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42561 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.222.62:42561: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
I20260924 00:50:40.146096 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:40.125497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.0.222.62:42561)
0924 00:50:40.126882 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:40.126888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:40.126905 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:50:40.126964 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:40.126969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:40.127037 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:40.127232 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:40.128799 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.129923 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.131511 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.132131 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.132258 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.135570 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:40.135583 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:40.135613 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:40.135669 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:40.141755 (+ 6086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:40.142683 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:40.142701 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:40.142711 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:40.142988 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:50:40.143435 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:50:40.143440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:50:40.143443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:50:40.144179 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:40.144526 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:40.145028 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 00:50:40.145658 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1273,"thread_start_us":1168,"threads_started":1}
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:40.149116 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:40.138831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:53149)
0924 00:50:40.139051 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:40.139070 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:40.139268 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:40.139716 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:40.139721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:40.140119 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:40.140305 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.140317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.142120 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.142125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.143341 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.143355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.144186 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.144192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.144362 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.145596 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:40.145620 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:40.148845 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 00:50:40.149310 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:40.139084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:53149 (local address 127.0.222.62:42561)
0924 00:50:40.139397 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:40.139404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:40.139422 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:50:40.139493 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:40.139498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:40.139577 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:40.139689 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:40.140457 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.141342 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.143497 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.144042 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.144153 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.149042 (+ 4889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:53149: BlockingRecv error: recv got EOF from 127.0.222.3:53149 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:40.159605 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:40.150012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:34009)
0924 00:50:40.150157 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:40.150174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:40.150276 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:40.150848 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:40.150852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:40.151137 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:40.151276 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.151284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.152542 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.152548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.154397 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.154414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.155138 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.155144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.155335 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.155974 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:40.155997 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:40.159386 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 00:50:40.159965 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:40.150348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:34009 (local address 127.0.222.62:42561)
0924 00:50:40.150592 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:40.150596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:40.150610 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:50:40.150651 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:40.150655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:40.150714 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:40.150794 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:40.151400 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.152406 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.154595 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.155012 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.155101 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.159660 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:34009: BlockingRecv error: recv got EOF from 127.0.222.3:34009 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260924 00:50:40.160044 10240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:40.236987 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:40.228321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:37033)
0924 00:50:40.228829 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:40.228844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:40.228974 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:40.229378 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:40.229384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:40.229974 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:40.230226 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.230240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.231271 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.231277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.232236 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.232249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.232956 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.232962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.233129 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.233673 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:40.233690 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:40.236717 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":271,"threads_started":1}
W20260924 00:50:40.237323 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:40.228425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:37033 (local address 127.0.222.62:42561)
0924 00:50:40.228621 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:40.228628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:40.228923 (+ 295us) server_negotiation.cc:407] Connection header received
0924 00:50:40.229118 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:40.229123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:40.229228 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:40.229344 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:40.230387 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.231129 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.232377 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.232789 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.232895 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.237003 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:37033: BlockingRecv error: recv got EOF from 127.0.222.1:37033 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:40 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:40.688583 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:40.677818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:54547)
0924 00:50:40.678340 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:40.678360 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:40.678514 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:40.678939 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:40.678945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:40.679583 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:40.679806 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.679817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.681042 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.681049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.682764 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:40.682783 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.683617 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.683623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:40.683813 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.684469 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:40.684493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:40.688259 (+ 3766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":253,"threads_started":1}
W20260924 00:50:40.688794 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:40.677865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:54547 (local address 127.0.222.62:42561)
0924 00:50:40.678096 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:40.678103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:40.678448 (+ 345us) server_negotiation.cc:407] Connection header received
0924 00:50:40.678674 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:40.678680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:40.678766 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:40.678873 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:40.679974 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.680859 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.682969 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:40.683439 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:40.683544 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:40.688521 (+ 4977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:54547: BlockingRecv error: recv got EOF from 127.0.222.2:54547 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 00:50:41 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:41 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:50:41.169406 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:41.160783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:33307 (local address 127.0.222.62:42561)
0924 00:50:41.160949 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:41.160953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:41.161346 (+ 393us) server_negotiation.cc:407] Connection header received
0924 00:50:41.161483 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:41.161487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:41.161558 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:41.161671 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:41.162566 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.163356 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.164714 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.165249 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.165346 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:41.169123 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:33307: BlockingRecv error: recv got EOF from 127.0.222.3:33307 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:50:41.169476 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:41.160698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:33307)
0924 00:50:41.161250 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:41.161266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:41.161381 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:41.161686 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:41.161689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:41.162201 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:41.162420 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:41.162430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.163507 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.163511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:41.164543 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:41.164557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.165443 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.165448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:41.165577 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:41.166119 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:41.166137 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:41.169152 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":269,"threads_started":1}
Sep 24 00:50:41 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:41 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:41.247786 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:41.238175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:55679)
0924 00:50:41.238788 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:41.238807 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:41.238940 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:41.239240 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:41.239245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:41.239632 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:41.239848 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:41.239858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.241055 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.241059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:41.242062 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:41.242077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.243019 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.243026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:41.243272 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:41.244159 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:41.244184 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:41.247499 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":342,"threads_started":1}
W20260924 00:50:41.248121 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:41.238343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55679 (local address 127.0.222.62:42561)
0924 00:50:41.238631 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:41.238637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:41.238894 (+ 257us) server_negotiation.cc:407] Connection header received
0924 00:50:41.239054 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:41.239058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:41.239124 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:41.239209 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:41.240059 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.240902 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.242208 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.242811 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.242944 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:41.247850 (+ 4906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:55679: BlockingRecv error: recv got EOF from 127.0.222.1:55679 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 24 00:50:41 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:41 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:41.699051 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:41.689877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:37973)
0924 00:50:41.690440 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:41.690457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:41.690602 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:41.690929 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:41.690933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:41.691346 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:41.691509 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:41.691523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.692814 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.692819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:41.694114 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:41.694129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.694960 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.694967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:41.695085 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:41.695714 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:41.695735 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:41.698763 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":325,"threads_started":1}
W20260924 00:50:41.699255 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:41.689954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:37973 (local address 127.0.222.62:42561)
0924 00:50:41.690112 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:41.690116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:41.690589 (+ 473us) server_negotiation.cc:407] Connection header received
0924 00:50:41.690720 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:41.690724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:41.690797 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:41.690896 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:41.691784 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.692681 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.694268 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:41.694801 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:41.694908 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:41.699014 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:37973: BlockingRecv error: recv got EOF from 127.0.222.2:37973 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 00:50:42 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:42 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:42.179041 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:42.170424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:49299)
0924 00:50:42.171032 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:42.171048 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:42.171171 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:42.171449 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:42.171452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:42.171892 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:42.172041 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:42.172050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.173157 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.173162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:42.174088 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:42.174101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.174906 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.174911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:42.175032 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:42.175651 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:42.175747 (+ 96us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:42.178701 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":380,"threads_started":1}
W20260924 00:50:42.179280 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:42.170571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:49299 (local address 127.0.222.62:42561)
0924 00:50:42.170791 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:42.170797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:42.171155 (+ 358us) server_negotiation.cc:407] Connection header received
0924 00:50:42.171267 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:42.171271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:42.171336 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:42.171420 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:42.172192 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.173013 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.174227 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.174738 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.174844 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:42.179018 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:49299: BlockingRecv error: recv got EOF from 127.0.222.3:49299 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 00:50:42 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:42 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:42.257248 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:42.248956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:60191)
0924 00:50:42.249513 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:42.249530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:42.249671 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:42.249971 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:42.249974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:42.250355 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:42.250537 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:42.250546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.251618 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.251624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:42.252575 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:42.252588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.253321 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.253335 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:42.253464 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:42.254067 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:42.254084 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:42.256891 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":316,"threads_started":1}
W20260924 00:50:42.257396 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:42.249054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:60191 (local address 127.0.222.62:42561)
0924 00:50:42.249338 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:42.249344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:42.249635 (+ 291us) server_negotiation.cc:407] Connection header received
0924 00:50:42.249778 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:42.249781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:42.249846 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:42.249930 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:42.250677 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.251471 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.252733 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.253139 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.253312 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:42.257104 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:60191: BlockingRecv error: recv got EOF from 127.0.222.1:60191 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 00:50:42 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:42 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:42.708937 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:42.700225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:45667)
0924 00:50:42.700770 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:42.700786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:42.700928 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:42.701287 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:42.701293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:42.701639 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:42.701824 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:42.701833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.702880 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.702885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:42.703794 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:42.703806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.704503 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.704509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:42.704678 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:42.705322 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:42.705340 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:42.708515 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":263,"threads_started":1}
W20260924 00:50:42.708969 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:42.700370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45667 (local address 127.0.222.62:42561)
0924 00:50:42.700581 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:42.700588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:42.700873 (+ 285us) server_negotiation.cc:407] Connection header received
0924 00:50:42.701048 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:42.701054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:42.701126 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:42.701248 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:42.701976 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.702735 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.703935 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:42.704358 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:42.704470 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:42.708762 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:45667: BlockingRecv error: recv got EOF from 127.0.222.2:45667 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 00:50:43 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:43 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:43.188858 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:43.180266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:33265)
0924 00:50:43.180793 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:43.180813 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:43.180936 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:43.181313 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:43.181316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:43.181763 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:43.181933 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:43.181943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.182954 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.182959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:43.184390 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:43.184405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.185085 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.185091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:43.185248 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:43.185776 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:43.185792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:43.188564 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":317,"threads_started":1}
W20260924 00:50:43.189327 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:43.180421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:33265 (local address 127.0.222.62:42561)
0924 00:50:43.180581 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:43.180586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:43.180902 (+ 316us) server_negotiation.cc:407] Connection header received
0924 00:50:43.181048 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:43.181053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:43.181153 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:43.181279 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:43.182067 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.182816 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.184532 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.184943 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.185033 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:43.189031 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:33265: BlockingRecv error: recv got EOF from 127.0.222.3:33265 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 00:50:43 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:43 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:43.266445 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:43.258322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:38305)
0924 00:50:43.258767 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:43.258783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:43.258891 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:43.259173 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:43.259177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:43.259548 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:43.259758 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:43.259770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.260770 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.260774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:43.261807 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:43.261819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.262513 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.262518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:43.262659 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:43.263268 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:43.263284 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:43.266141 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":235,"threads_started":1}
W20260924 00:50:43.266633 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:43.258411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:38305 (local address 127.0.222.62:42561)
0924 00:50:43.258578 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:43.258582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:43.258853 (+ 271us) server_negotiation.cc:407] Connection header received
0924 00:50:43.258978 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:43.258981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:43.259044 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:43.259127 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:43.259878 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.260616 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.261955 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.262375 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.262466 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:43.266384 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:38305: BlockingRecv error: recv got EOF from 127.0.222.1:38305 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 00:50:43 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:43 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:43.718386 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:43.709975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:46817)
0924 00:50:43.710509 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:43.710524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:43.710661 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:43.711059 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:43.711063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:43.711456 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:43.711605 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:43.711614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.712581 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.712586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:43.713727 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:43.713742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.714505 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.714511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:43.714691 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:43.715278 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:43.715295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:43.718141 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":301,"threads_started":1}
W20260924 00:50:43.718540 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:43.710216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:46817 (local address 127.0.222.62:42561)
0924 00:50:43.710444 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:43.710450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:43.710602 (+ 152us) server_negotiation.cc:407] Connection header received
0924 00:50:43.710853 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:43.710857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:43.710925 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:43.711005 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:43.711751 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.712455 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.713873 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:43.714357 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:43.714477 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:43.718345 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:46817: BlockingRecv error: recv got EOF from 127.0.222.2:46817 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 00:50:44 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:44 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:44.198833 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:44.189883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:46527)
0924 00:50:44.190366 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:44.190384 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:44.190540 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:44.190919 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:44.190922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:44.191398 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:44.191602 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:44.191616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.193045 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.193050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:44.194088 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:44.194101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.194801 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.194807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:44.194940 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:44.195588 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:44.195604 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:44.198517 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
W20260924 00:50:44.199011 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:44.189994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:46527 (local address 127.0.222.62:42561)
0924 00:50:44.190158 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:44.190162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:44.190465 (+ 303us) server_negotiation.cc:407] Connection header received
0924 00:50:44.190653 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:44.190657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:44.190751 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:44.190865 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:44.191770 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.192902 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.194242 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.194664 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.194759 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:44.198766 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:46527: BlockingRecv error: recv got EOF from 127.0.222.3:46527 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 00:50:44 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:44 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:44.275593 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:44.267605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:58365)
0924 00:50:44.268106 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:44.268121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:44.268239 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:44.268505 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:44.268508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:44.268882 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:44.269052 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:44.269061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.270087 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.270091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:44.271018 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:44.271030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.271725 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.271731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:44.271865 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:44.272424 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:44.272441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:44.275299 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":290,"threads_started":1}
W20260924 00:50:44.275774 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:44.267685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:58365 (local address 127.0.222.62:42561)
0924 00:50:44.267901 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:44.267908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:44.268205 (+ 297us) server_negotiation.cc:407] Connection header received
0924 00:50:44.268334 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:44.268337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:44.268399 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:44.268475 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:44.269172 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.269951 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.271143 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.271573 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.271667 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:44.275530 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:58365: BlockingRecv error: recv got EOF from 127.0.222.1:58365 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 00:50:44 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:44 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:44.731392 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:44.719617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:55441)
0924 00:50:44.720190 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:44.720205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:44.720327 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:44.720618 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:44.720621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:44.721058 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:44.721267 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:44.721279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.722411 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.722415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:44.723339 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:44.723350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.724159 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.724165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:44.724326 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:44.728184 (+ 3858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:44.728237 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:44.731138 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":307,"threads_started":1}
W20260924 00:50:44.731673 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:44.719976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:55441 (local address 127.0.222.62:42561)
0924 00:50:44.720153 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:44.720157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:44.720321 (+ 164us) server_negotiation.cc:407] Connection header received
0924 00:50:44.720425 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:44.720429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:44.720496 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:44.720587 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:44.721434 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.722286 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.723493 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:44.723947 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:44.724077 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:44.731405 (+ 7328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:55441: BlockingRecv error: recv got EOF from 127.0.222.2:55441 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 00:50:45 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:45 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:45.207855 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:45.199738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:43579)
0924 00:50:45.200242 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:45.200258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:45.200370 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:45.200658 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:45.200661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:45.201069 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:45.201277 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:45.201291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.202358 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.202361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:45.203318 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:45.203331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.203972 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.203976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:45.204091 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:45.204616 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:45.204632 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:45.207577 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":235,"threads_started":1}
W20260924 00:50:45.208069 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:45.199876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:43579 (local address 127.0.222.62:42561)
0924 00:50:45.200025 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:45.200029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:45.200338 (+ 309us) server_negotiation.cc:407] Connection header received
0924 00:50:45.200464 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:45.200468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:45.200546 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:45.200626 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:45.201421 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.202246 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.203450 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.203860 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.203959 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:45.207817 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:43579: BlockingRecv error: recv got EOF from 127.0.222.3:43579 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 00:50:45 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:45 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:45.284747 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:45.276602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:49997)
0924 00:50:45.277144 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:45.277159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:45.277354 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:45.277668 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:45.277673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:45.278118 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:45.278279 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:45.278288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.279278 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.279283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:45.280161 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:45.280173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.280866 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.280871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:45.281041 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:45.281667 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:45.281684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:45.284471 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":317,"threads_started":1}
W20260924 00:50:45.284911 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:45.276731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:49997 (local address 127.0.222.62:42561)
0924 00:50:45.276967 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:45.276974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:45.277292 (+ 318us) server_negotiation.cc:407] Connection header received
0924 00:50:45.277472 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:45.277478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:45.277544 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:45.277624 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:45.278419 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.279143 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.280309 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.280717 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.280824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:45.284676 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:49997: BlockingRecv error: recv got EOF from 127.0.222.1:49997 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 00:50:45 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:45 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:45.740407 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:45.732271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:50193)
0924 00:50:45.732801 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:45.732817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:45.732953 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:45.733301 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:45.733306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:45.733706 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:45.733869 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:45.733878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.734876 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.734879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:45.735837 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:45.735850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.736572 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.736578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:45.736739 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:45.737369 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:45.737386 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:45.740084 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":288,"threads_started":1}
W20260924 00:50:45.740589 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:45.732362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50193 (local address 127.0.222.62:42561)
0924 00:50:45.732592 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:45.732598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:45.732909 (+ 311us) server_negotiation.cc:407] Connection header received
0924 00:50:45.733061 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:45.733066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:45.733137 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:45.733254 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:45.734029 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.734744 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.735976 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:45.736419 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:45.736528 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:45.740359 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:50193: BlockingRecv error: recv got EOF from 127.0.222.2:50193 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 00:50:46 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:46 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:46.217896 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:46.208853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:34763)
0924 00:50:46.209493 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:46.209514 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:46.209660 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:46.210084 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:46.210088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:46.210478 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:46.210739 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:46.210750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.211817 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.211821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:46.212716 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:46.212728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.213510 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.213515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:46.213654 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:46.214414 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:46.214433 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:46.217596 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":354,"threads_started":1}
W20260924 00:50:46.218109 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:46.209018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:34763 (local address 127.0.222.62:42561)
0924 00:50:46.209324 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:46.209330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:46.209609 (+ 279us) server_negotiation.cc:407] Connection header received
0924 00:50:46.209851 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:46.209855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:46.209922 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:46.210002 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:46.210886 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.211634 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.212855 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.213338 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.213439 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:46.217869 (+ 4430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:34763: BlockingRecv error: recv got EOF from 127.0.222.3:34763 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 24 00:50:46 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:46 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:46.294867 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:46.285917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:36669)
0924 00:50:46.286552 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:46.286568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:46.286760 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:46.287108 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:46.287112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:46.287636 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:46.287850 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:46.287865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.289298 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.289302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:46.290213 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:46.290225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.290888 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.290892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:46.291016 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:46.291596 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:46.291613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:46.294571 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":287,"threads_started":1}
W20260924 00:50:46.295081 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:46.286074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:36669 (local address 127.0.222.62:42561)
0924 00:50:46.286341 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:46.286348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:46.286663 (+ 315us) server_negotiation.cc:407] Connection header received
0924 00:50:46.286894 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:46.286899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:46.286962 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:46.287059 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:46.288065 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.289092 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.290357 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.290756 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.290852 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:46.294855 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:36669: BlockingRecv error: recv got EOF from 127.0.222.1:36669 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 00:50:46 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:46 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:46.749703 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:46.741454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:37627)
0924 00:50:46.742010 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:46.742025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:46.742135 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:46.742417 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:46.742420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:46.742830 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:46.743011 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:46.743020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.744024 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.744030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:46.744941 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:46.744953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.745708 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.745714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:46.745829 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:46.746327 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:46.746344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:46.749390 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":254,"threads_started":1}
W20260924 00:50:46.749835 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:46.741538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:37627 (local address 127.0.222.62:42561)
0924 00:50:46.741848 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:46.741854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:46.742093 (+ 239us) server_negotiation.cc:407] Connection header received
0924 00:50:46.742219 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:46.742222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:46.742308 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:46.742390 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:46.743123 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.743893 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.745063 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:46.745570 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:46.745668 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:46.749589 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:37627: BlockingRecv error: recv got EOF from 127.0.222.2:37627 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 24 00:50:47 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:47 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:47.227356 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:47.218933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:49185)
0924 00:50:47.219552 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:47.219574 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:47.219723 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:47.220079 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:47.220084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:47.220579 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:47.220793 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:47.220803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.221903 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.221907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:47.222826 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:47.222838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.223498 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.223502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:47.223605 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:47.224164 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:47.224180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:47.227113 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":314,"threads_started":1}
W20260924 00:50:47.227671 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:47.219059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:49185 (local address 127.0.222.62:42561)
0924 00:50:47.219273 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:47.219278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:47.219691 (+ 413us) server_negotiation.cc:407] Connection header received
0924 00:50:47.219831 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:47.219836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:47.219915 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:47.220008 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:47.220936 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.221780 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.222951 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.223388 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.223476 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:47.227365 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:49185: BlockingRecv error: recv got EOF from 127.0.222.3:49185 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 00:50:47 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:47 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:47.304054 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:47.295831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:32955)
0924 00:50:47.296333 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:47.296348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:47.296478 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:47.296813 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:47.296818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:47.297260 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:47.297414 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:47.297423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.298422 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.298427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:47.299373 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:47.299386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.300104 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.300110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:47.300283 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:47.300932 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:47.300950 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:47.303717 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":225,"threads_started":1}
W20260924 00:50:47.304248 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:47.296036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:32955 (local address 127.0.222.62:42561)
0924 00:50:47.296236 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:47.296242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:47.296429 (+ 187us) server_negotiation.cc:407] Connection header received
0924 00:50:47.296585 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:47.296590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:47.296658 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:47.296736 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:47.297550 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.298290 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.299506 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.299928 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.300041 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:47.303999 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:32955: BlockingRecv error: recv got EOF from 127.0.222.1:32955 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 00:50:47 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:47 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:47.758953 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:47.750679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:50153)
0924 00:50:47.751157 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:47.751174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:47.751304 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:47.751589 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:47.751592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:47.752022 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:47.752184 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:47.752193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.753343 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.753348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:47.754315 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:47.754328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.755117 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.755123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:47.755295 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:47.755906 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:47.755929 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:47.758717 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
W20260924 00:50:47.759132 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:47.750743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50153 (local address 127.0.222.62:42561)
0924 00:50:47.750952 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:47.750958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:47.751243 (+ 285us) server_negotiation.cc:407] Connection header received
0924 00:50:47.751397 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:47.751401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:47.751465 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:47.751542 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:47.752300 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.753164 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.754435 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:47.754947 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:47.755049 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:47.758914 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:50153: BlockingRecv error: recv got EOF from 127.0.222.2:50153 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 00:50:48 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:48 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:48.236955 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:48.228293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:35445)
0924 00:50:48.228897 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:48.228914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:48.229049 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:48.229400 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:48.229405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:48.229884 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:48.230036 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:48.230046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.231196 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.231199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:48.232126 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:48.232138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.232879 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.232885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:48.233040 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:48.233621 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:48.233638 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:48.236645 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":279,"threads_started":1}
W20260924 00:50:48.237186 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:48.228444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35445 (local address 127.0.222.62:42561)
0924 00:50:48.228711 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:48.228726 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:48.229000 (+ 274us) server_negotiation.cc:407] Connection header received
0924 00:50:48.229166 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:48.229172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:48.229263 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:48.229344 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:48.230218 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.231036 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.232266 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.232701 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.232818 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:48.236885 (+ 4067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:35445: BlockingRecv error: recv got EOF from 127.0.222.3:35445 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 24 00:50:48 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:48 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:48.313299 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:48.305257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:59009)
0924 00:50:48.305740 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:48.305756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:48.305878 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:48.306216 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:48.306218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:48.306628 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:48.306790 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:48.306799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.307791 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.307795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:48.308713 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:48.308723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.309395 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.309399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:48.309527 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:48.310047 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:48.310063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:48.313019 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":237,"threads_started":1}
W20260924 00:50:48.313474 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:48.305348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59009 (local address 127.0.222.62:42561)
0924 00:50:48.305587 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:48.305594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:48.305839 (+ 245us) server_negotiation.cc:407] Connection header received
0924 00:50:48.306030 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:48.306034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:48.306102 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:48.306221 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:48.306916 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.307653 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.308826 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.309271 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.309359 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:48.313267 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:59009: BlockingRecv error: recv got EOF from 127.0.222.1:59009 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 00:50:48 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:48 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:48.768476 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:48.759868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:49287)
0924 00:50:48.760397 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:48.760414 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:48.760534 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:48.760927 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:48.760931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:48.761385 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:48.761554 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:48.761564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.762567 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.762571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:48.763584 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:48.763598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.764524 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.764530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:48.764702 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:48.765344 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:48.765364 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:48.768226 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":274,"threads_started":1}
W20260924 00:50:48.768689 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:48.759981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:49287 (local address 127.0.222.62:42561)
0924 00:50:48.760164 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:48.760168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:48.760510 (+ 342us) server_negotiation.cc:407] Connection header received
0924 00:50:48.760699 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:48.760704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:48.760776 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:48.760865 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:48.761716 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.762463 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.763751 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:48.764348 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:48.764509 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:48.768458 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:49287: BlockingRecv error: recv got EOF from 127.0.222.2:49287 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 00:50:49 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:49 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:49.246623 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:49.238064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:35793)
0924 00:50:49.238629 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:49.238644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:49.238772 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:49.239065 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:49.239070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:49.239576 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:49.239787 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:49.239800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.240810 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.240815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:49.241907 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:49.241920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.242622 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.242627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:49.242797 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:49.243338 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:49.243355 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:49.246280 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":332,"threads_started":1}
W20260924 00:50:49.246785 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:49.238170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35793 (local address 127.0.222.62:42561)
0924 00:50:49.238351 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:49.238358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:49.238724 (+ 366us) server_negotiation.cc:407] Connection header received
0924 00:50:49.238883 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:49.238889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:49.238955 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:49.239040 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:49.239932 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.240680 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.242041 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.242462 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.242594 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:49.246547 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:35793: BlockingRecv error: recv got EOF from 127.0.222.3:35793 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 00:50:49 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:49 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:49.322074 10272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:49.314231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:59245)
0924 00:50:49.314719 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:49.314735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:49.314859 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:49.315177 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:49.315181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:49.315535 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:49.315683 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:49.315691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.316629 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.316632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:49.317664 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:49.317677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.318356 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.318361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:49.318484 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:49.319085 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:49.319103 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:49.321836 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":244,"threads_started":1}
W20260924 00:50:49.322249 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:49.314409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59245 (local address 127.0.222.62:42561)
0924 00:50:49.314649 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:49.314655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:49.314812 (+ 157us) server_negotiation.cc:407] Connection header received
0924 00:50:49.314943 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:49.314947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:49.315010 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:49.315087 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:49.315809 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.316532 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.317801 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.318229 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.318323 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:49.322023 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:59245: BlockingRecv error: recv got EOF from 127.0.222.1:59245 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 00:50:49 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:49 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:49.783433 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:49.769558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:45153)
0924 00:50:49.770135 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:49.770184 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:49.770326 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:49.770666 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:49.770671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:49.771047 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:49.771205 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:49.771214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.772240 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.772244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:49.773172 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:49.773186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.774086 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.774095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:49.774302 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:49.780050 (+ 5748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:49.780070 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:49.783178 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":315,"threads_started":1}
W20260924 00:50:49.783776 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:49.769667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45153 (local address 127.0.222.62:42561)
0924 00:50:49.769976 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:49.769982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:49.770277 (+ 295us) server_negotiation.cc:407] Connection header received
0924 00:50:49.770463 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:49.770467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:49.770531 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:49.770614 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:49.771354 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.772090 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.773358 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:49.773843 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:49.773971 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:49.783434 (+ 9463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:45153: BlockingRecv error: recv got EOF from 127.0.222.2:45153 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
Sep 24 00:50:50 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:50 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:50.255776 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:50.247662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:44037)
0924 00:50:50.248173 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:50.248211 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:50.248318 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:50.248649 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:50.248653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:50.249011 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:50.249174 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:50.249182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.250249 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.250254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:50.251201 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:50.251213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.251887 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.251893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:50.252068 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:50.252606 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:50.252628 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:50.255521 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":291,"threads_started":1}
W20260924 00:50:50.256085 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:50.247913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:44037 (local address 127.0.222.62:42561)
0924 00:50:50.248086 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:50.248091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:50.248268 (+ 177us) server_negotiation.cc:407] Connection header received
0924 00:50:50.248425 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:50.248429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:50.248491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:50.248563 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:50.249341 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.250101 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.251330 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.251739 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.251857 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:50.255814 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:44037: BlockingRecv error: recv got EOF from 127.0.222.3:44037 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 00:50:50 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:50 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:50.331552 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:50.323058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:57219)
0924 00:50:50.323591 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:50.323607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:50.323750 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:50.324151 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:50.324156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:50.324575 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:50.324774 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:50.324785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.325937 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.325944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:50.326907 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:50.326920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.327685 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.327692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:50.327868 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:50.328458 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:50.328477 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:50.331297 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":266,"threads_started":1}
W20260924 00:50:50.331740 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:50.323172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:57219 (local address 127.0.222.62:42561)
0924 00:50:50.323436 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:50.323442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:50.323693 (+ 251us) server_negotiation.cc:407] Connection header received
0924 00:50:50.323879 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:50.323885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:50.323970 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:50.324057 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:50.324934 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.325747 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.327054 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.327496 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.327632 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:50.331534 (+ 3902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:57219: BlockingRecv error: recv got EOF from 127.0.222.1:57219 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 00:50:50 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:50 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:50.793320 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:50.784430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:35989)
0924 00:50:50.785002 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:50.785019 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:50.785158 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:50.785579 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:50.785583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:50.785986 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:50.786187 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:50.786197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.787395 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.787399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:50.788422 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:50.788437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.789221 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.789227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:50.789363 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:50.789940 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:50.789957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:50.792988 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":328,"threads_started":1}
W20260924 00:50:50.793476 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:50.784582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:35989 (local address 127.0.222.62:42561)
0924 00:50:50.784810 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:50.784817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:50.785128 (+ 311us) server_negotiation.cc:407] Connection header received
0924 00:50:50.785337 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:50.785342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:50.785426 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:50.785510 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:50.786361 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.787229 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.788603 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:50.789035 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:50.789141 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:50.793250 (+ 4109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:35989: BlockingRecv error: recv got EOF from 127.0.222.2:35989 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 00:50:51 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:51 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:51.265569 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:51.256705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:45339)
0924 00:50:51.257236 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:51.257257 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:51.257383 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:51.257664 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:51.257667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:51.258073 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:51.258286 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:51.258295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.259384 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.259390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:51.260604 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:51.260620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.261559 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.261565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:51.261713 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:51.262327 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:51.262346 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:51.265312 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
W20260924 00:50:51.265765 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:51.256837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:45339 (local address 127.0.222.62:42561)
0924 00:50:51.257104 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:51.257111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:51.257338 (+ 227us) server_negotiation.cc:407] Connection header received
0924 00:50:51.257476 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:51.257480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:51.257547 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:51.257630 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:51.258414 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.259198 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.260740 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.261408 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.261516 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:51.265539 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:45339: BlockingRecv error: recv got EOF from 127.0.222.3:45339 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 24 00:50:51 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:51 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:51.341300 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:51.332640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:45341)
0924 00:50:51.333164 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:51.333179 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:51.333359 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:51.333687 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:51.333692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:51.334162 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:51.334321 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:51.334331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.335562 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.335567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:51.336668 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:51.336682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.337438 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.337443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:51.337583 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:51.338208 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:51.338226 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:51.340989 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":280,"threads_started":1}
W20260924 00:50:51.341658 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:51.332774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45341 (local address 127.0.222.62:42561)
0924 00:50:51.333020 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:51.333026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:51.333326 (+ 300us) server_negotiation.cc:407] Connection header received
0924 00:50:51.333485 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:51.333491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:51.333565 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:51.333646 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:51.334495 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.335434 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.336791 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.337304 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.337399 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:51.341234 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:45341: BlockingRecv error: recv got EOF from 127.0.222.1:45341 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 00:50:51 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:51 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:51.802465 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:51.794342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:49947)
0924 00:50:51.794826 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:51.794846 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:51.794974 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:51.795279 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:51.795282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:51.795690 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:51.795840 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:51.795848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.796860 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.796864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:51.797845 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:51.797857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.798571 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.798576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:51.798725 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:51.799272 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:51.799288 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:51.802229 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":283,"threads_started":1}
W20260924 00:50:51.802639 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:51.794418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:49947 (local address 127.0.222.62:42561)
0924 00:50:51.794632 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:51.794637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:51.795010 (+ 373us) server_negotiation.cc:407] Connection header received
0924 00:50:51.795073 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:51.795078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:51.795162 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:51.795240 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:51.795969 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.796727 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.797995 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:51.798437 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:51.798536 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:51.802437 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:49947: BlockingRecv error: recv got EOF from 127.0.222.2:49947 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 00:50:52 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:52 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:52.275100 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:52.266575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:37247)
0924 00:50:52.267157 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:52.267176 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:52.267312 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:52.267639 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:52.267643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:52.268014 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:52.268169 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:52.268178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.269270 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.269274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:52.270236 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:52.270249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.270978 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.270983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:52.271159 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:52.271881 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:52.271899 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:52.274864 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":352,"threads_started":1}
W20260924 00:50:52.275377 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:52.266673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:37247 (local address 127.0.222.62:42561)
0924 00:50:52.266927 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:52.266934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:52.267243 (+ 309us) server_negotiation.cc:407] Connection header received
0924 00:50:52.267432 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:52.267435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:52.267514 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:52.267604 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:52.268331 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.269082 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.270391 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.270832 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.270927 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:52.275101 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:37247: BlockingRecv error: recv got EOF from 127.0.222.3:37247 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 00:50:52 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:52 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:52.350756 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:52.342343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:55085)
0924 00:50:52.342865 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:52.342880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:52.343005 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:52.343324 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:52.343329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:52.343720 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:52.343905 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:52.343914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.344959 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.344963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:52.345918 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:52.345930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.346606 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.346611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:52.346764 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:52.347330 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:52.347349 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:52.350451 (+ 3102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
W20260924 00:50:52.350941 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:52.342391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55085 (local address 127.0.222.62:42561)
0924 00:50:52.342587 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:52.342594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:52.342961 (+ 367us) server_negotiation.cc:407] Connection header received
0924 00:50:52.343109 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:52.343114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:52.343178 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:52.343253 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:52.344036 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.344830 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.346046 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.346469 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.346573 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:52.350692 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:55085: BlockingRecv error: recv got EOF from 127.0.222.1:55085 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 00:50:52 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:52 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:52.811846 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:52.803440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:38561)
0924 00:50:52.803947 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:52.803961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:52.804088 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:52.804397 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:52.804401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:52.804791 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:52.804943 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:52.804952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.806024 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.806028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:52.806961 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:52.806973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.807677 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.807684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:52.807832 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:52.808373 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:52.808392 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:52.811607 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":237,"threads_started":1}
W20260924 00:50:52.812058 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:52.803627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:38561 (local address 127.0.222.62:42561)
0924 00:50:52.803883 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:52.803889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:52.804032 (+ 143us) server_negotiation.cc:407] Connection header received
0924 00:50:52.804197 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:52.804203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:52.804270 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:52.804349 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:52.805080 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.805890 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.807096 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:52.807495 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:52.807603 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:52.811819 (+ 4216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:38561: BlockingRecv error: recv got EOF from 127.0.222.2:38561 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 00:50:53 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:53 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:53.285179 10283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:53.276029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:37599)
0924 00:50:53.276630 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:53.276651 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:53.276801 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:53.277152 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:53.277158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:53.277639 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:53.277820 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:53.277831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.278898 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.278903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:53.279884 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:53.279898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.280698 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.280705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:53.280888 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:53.281545 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:53.281567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:53.284829 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":296,"threads_started":1}
W20260924 00:50:53.285383 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:53.276156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:37599 (local address 127.0.222.62:42561)
0924 00:50:53.276367 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:53.276374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:53.276748 (+ 374us) server_negotiation.cc:407] Connection header received
0924 00:50:53.276929 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:53.276934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:53.277005 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:53.277103 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:53.277977 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.278746 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.280042 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.280500 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.280633 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:53.285033 (+ 4400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:37599: BlockingRecv error: recv got EOF from 127.0.222.3:37599 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 00:50:53 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:53 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:53.370770 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:53.352008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:50053)
0924 00:50:53.353335 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:53.353373 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:53.353599 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:53.353938 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:53.353941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:53.354307 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:53.354509 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:53.354518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.355572 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.355575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:53.357635 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:53.357672 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.359545 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.359555 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:53.359916 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:53.361458 (+ 1542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:53.361510 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:53.369955 (+ 8445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":799,"threads_started":1}
W20260924 00:50:53.371224 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:53.352157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:50053 (local address 127.0.222.62:42561)
0924 00:50:53.353271 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:53.353334 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:53.353489 (+ 155us) server_negotiation.cc:407] Connection header received
0924 00:50:53.353753 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:53.353763 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:53.353830 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:53.353911 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:53.354691 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.355456 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.357976 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.359237 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.359461 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:53.370658 (+ 11197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:50053: BlockingRecv error: recv got EOF from 127.0.222.1:50053 (error 108)
Metrics: {"server-negotiator.queue_time_us":448}
Sep 24 00:50:53 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:53 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:53.821033 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:53.812874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:50979)
0924 00:50:53.813419 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:53.813441 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:53.813586 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:53.813903 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:53.813906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:53.814318 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:53.814517 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:53.814526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.815577 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.815581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:53.816518 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:53.816531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.817183 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.817187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:53.817321 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:53.817924 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:53.817942 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:53.820685 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":310,"threads_started":1}
W20260924 00:50:53.821164 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:53.812926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50979 (local address 127.0.222.62:42561)
0924 00:50:53.813092 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:53.813096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:53.813523 (+ 427us) server_negotiation.cc:407] Connection header received
0924 00:50:53.813682 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:53.813686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:53.813779 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:53.813872 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:53.814652 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.815441 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.816638 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:53.817053 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:53.817142 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:53.820930 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:50979: BlockingRecv error: recv got EOF from 127.0.222.2:50979 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 00:50:54 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:54 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.3@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:54.294915 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:54.286428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.3:54251)
0924 00:50:54.287009 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:54.287025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:54.287157 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:54.287507 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:54.287512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:54.287933 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:54.288118 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:54.288128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.289113 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.289117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:54.290074 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:54.290086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.290769 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.290775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:54.290931 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:54.291537 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:54.291560 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:54.294583 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":363,"threads_started":1}
W20260924 00:50:54.295121 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:54.286557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:54251 (local address 127.0.222.62:42561)
0924 00:50:54.286755 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:54.286761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:54.287110 (+ 349us) server_negotiation.cc:407] Connection header received
0924 00:50:54.287293 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:54.287297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:54.287376 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:54.287457 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:54.288266 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.288976 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.290201 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.290630 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.290736 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:54.294879 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.3:54251: BlockingRecv error: recv got EOF from 127.0.222.3:54251 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 00:50:54 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:54 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.1@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:54.382686 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:54.373045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.1:50837)
0924 00:50:54.373691 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:54.373708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:54.373835 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:54.374170 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:54.374174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:54.374675 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:54.374910 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:54.374919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.376074 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.376079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:54.377181 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:54.377220 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.378030 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.378035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:54.378163 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:54.378828 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:54.378847 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:54.382260 (+ 3413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":354,"threads_started":1}
W20260924 00:50:54.382822 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:54.373101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:50837 (local address 127.0.222.62:42561)
0924 00:50:54.373384 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:54.373391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:54.373790 (+ 399us) server_negotiation.cc:407] Connection header received
0924 00:50:54.373931 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:54.373935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:54.374024 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:54.374105 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:54.375061 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.375908 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.377368 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.377873 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.378011 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:54.382547 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.1:50837: BlockingRecv error: recv got EOF from 127.0.222.1:50837 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 00:50:54 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:50:54 dist-test-slave-rtvz krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.222.2@KRBTEST.COM for oryx/127.0.222.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:50:54.835758 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:54.822249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42561 (local address 127.0.222.2:38057)
0924 00:50:54.822860 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:54.822881 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:54.823025 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:54.823378 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:54.823383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:54.823845 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:54.824087 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:54.824101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.825537 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.825542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:54.826805 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:54.826824 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.828198 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.828204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:54.828402 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:54.832326 (+ 3924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:50:54.832348 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:54.835379 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.222.62:42561: Server oryx/127.0.222.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":332,"threads_started":1}
W20260924 00:50:54.835892 9976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:54.822377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:38057 (local address 127.0.222.62:42561)
0924 00:50:54.822627 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:54.822658 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:54.822951 (+ 293us) server_negotiation.cc:407] Connection header received
0924 00:50:54.823145 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:54.823150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:54.823232 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:54.823329 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:54.824255 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.825364 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.827019 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:54.827759 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:54.827892 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:54.835649 (+ 7757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.222.2:38057: BlockingRecv error: recv got EOF from 127.0.222.2:38057 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:50:55.131111 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9823
I20260924 00:50:55.144423 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:55.132233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.0.222.1:40125)
0924 00:50:55.132869 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:55.132876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:55.132901 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:50:55.132961 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:55.132967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:55.133179 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:55.133378 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:55.134787 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:55.135980 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:55.137658 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:55.138304 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:55.138411 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:55.140946 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:55.140959 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:55.140978 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:55.141016 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:55.143864 (+ 2848us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 00:50:55.144149 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39582: Request ticket server kudu/127.0.222.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":297,"threads_started":1}
W20260924 00:50:55.144524 10290 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39582: Request ticket server kudu/127.0.222.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 00:50:55.144827 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9823: Not authorized: Client connection negotiation failed: client connection to 127.0.222.1:40125: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 00:50:55.145035 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 9823
I20260924 00:50:55.159250 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9977
I20260924 00:50:55.172001 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:55.160366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.0.222.2:45491)
0924 00:50:55.161044 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:55.161051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:55.161080 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:50:55.161139 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:55.161146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:55.161425 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:55.161590 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:55.162918 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:55.163957 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:55.165374 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:55.165918 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:55.166015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:55.168644 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:55.168653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:55.168668 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:55.168708 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:55.171450 (+ 2742us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 00:50:55.171746 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37616: Request ticket server kudu/127.0.222.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":345,"threads_started":1}
W20260924 00:50:55.172106 10291 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37616: Request ticket server kudu/127.0.222.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 00:50:55.172245 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9977: Not authorized: Client connection negotiation failed: client connection to 127.0.222.2:45491: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 00:50:55.172429 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 9977
I20260924 00:50:55.185395 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 10110
I20260924 00:50:55.197810 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:55.186427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.0.222.3:45227)
0924 00:50:55.187001 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:55.187007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:55.187031 (+ 24us) server_negotiation.cc:407] Connection header received
0924 00:50:55.187093 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:55.187099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:55.187313 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:55.187458 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:55.188849 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:55.190045 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:55.191338 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:55.191884 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:55.191978 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:55.194598 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:55.194606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:55.194620 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:50:55.194668 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:55.197134 (+ 2466us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 00:50:55.197472 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49634: Request ticket server kudu/127.0.222.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":299,"threads_started":1}
W20260924 00:50:55.197903 10292 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49634: Request ticket server kudu/127.0.222.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 00:50:55.197928 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 10110: Not authorized: Client connection negotiation failed: client connection to 127.0.222.3:45227: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.222.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 00:50:55.198097 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 10110
I20260924 00:50:55.210779 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 9754
I20260924 00:50:55.212052 9816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:55.332322 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 9754
2026-09-24T00:50:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21034 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T00:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:55Z Disabled control of system clock
I20260924 00:50:55.402047 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:40425
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:40425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 00:50:55.732123 10297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:55.732424 10297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:55.732502 10297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:55.739773 10297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:55.739895 10297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:55.739933 10297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:55.739996 10297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:55.740037 10297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:55.749431 10297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:40425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:40425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:55.750713 10297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:55.752511 10297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:55.764596 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:55.764551 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:55.764605 10305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:55.765185 10297 server_base.cc:1034] running on GCE node
I20260924 00:50:55.766085 10297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:55.767580 10297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:55.768824 10297 hybrid_clock.cc:648] HybridClock initialized: now 1790211055768760 us; error 46 us; skew 500 ppm
I20260924 00:50:55.769307 10297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:55.771829 10297 webserver.cc:460] Webserver started at http://127.0.222.62:43777/ using document root <none> and password file <none>
I20260924 00:50:55.772586 10297 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:55.772686 10297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:55.773015 10297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:55.776111 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "edde42ac7f914ad0806c9bbab60269e0"
format_stamp: "Formatted at 2026-09-24 00:50:55 on dist-test-slave-rtvz"
server_key: "1de54916cc5ff15a15d5df0b3f3499f7"
server_key_iv: "50a6ac75cf311b8ac43bf297b5727c43"
server_key_version: "encryptionkey@0"
I20260924 00:50:55.776926 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "edde42ac7f914ad0806c9bbab60269e0"
format_stamp: "Formatted at 2026-09-24 00:50:55 on dist-test-slave-rtvz"
server_key: "1de54916cc5ff15a15d5df0b3f3499f7"
server_key_iv: "50a6ac75cf311b8ac43bf297b5727c43"
server_key_version: "encryptionkey@0"
I20260924 00:50:55.782344 10297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:50:55.786149 10311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:55.787727 10297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:50:55.787926 10297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "edde42ac7f914ad0806c9bbab60269e0"
format_stamp: "Formatted at 2026-09-24 00:50:55 on dist-test-slave-rtvz"
server_key: "1de54916cc5ff15a15d5df0b3f3499f7"
server_key_iv: "50a6ac75cf311b8ac43bf297b5727c43"
server_key_version: "encryptionkey@0"
I20260924 00:50:55.788151 10297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:55.803467 10297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:55.804404 10297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:55.804692 10297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:55.824052 10297 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:40425
I20260924 00:50:55.824074 10363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:40425 every 8 connection(s)
I20260924 00:50:55.826054 10297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:55.831429 10364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:55.833541 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 10297
I20260924 00:50:55.833776 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:55.834738 888 external_mini_cluster.cc:1428] Setting key 37cf633ce675db703ffff521151eb3dd
I20260924 00:50:55.846927 10364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0: Bootstrap starting.
I20260924 00:50:55.851527 10364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:55.852725 10364 log.cc:824] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:55.855654 10364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0: No bootstrap required, opened a new log
I20260924 00:50:55.856563 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:55.837480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.0.222.62:40425)
0924 00:50:55.840870 (+ 3390us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:55.840890 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:55.841073 (+ 183us) server_negotiation.cc:407] Connection header received
0924 00:50:55.846287 (+ 5214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:55.846320 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:55.847029 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:55.847501 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:55.848171 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:55.850298 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:55.852147 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:55.852875 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:55.853070 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:55.854299 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:55.854307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:55.854323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:55.854874 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:55.854950 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:55.854972 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:55.855086 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:55.855300 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 00:50:55.855606 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2798,"thread_start_us":2492,"threads_started":1}
I20260924 00:50:55.863382 10364 raft_consensus.cc:348] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde42ac7f914ad0806c9bbab60269e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40425 } }
I20260924 00:50:55.863785 10364 raft_consensus.cc:374] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:55.863895 10364 raft_consensus.cc:729] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edde42ac7f914ad0806c9bbab60269e0, State: Initialized, Role: FOLLOWER
I20260924 00:50:55.864485 10364 consensus_queue.cc:260] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde42ac7f914ad0806c9bbab60269e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40425 } }
I20260924 00:50:55.864707 10364 raft_consensus.cc:388] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:55.864825 10364 raft_consensus.cc:482] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:55.864979 10364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:55.867254 10364 raft_consensus.cc:504] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde42ac7f914ad0806c9bbab60269e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40425 } }
I20260924 00:50:55.867750 10364 leader_election.cc:302] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edde42ac7f914ad0806c9bbab60269e0; no voters:
I20260924 00:50:55.868367 10364 leader_election.cc:288] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:55.868511 10369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:55.870151 10369 raft_consensus.cc:686] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [term 1 LEADER]: Becoming Leader. State: Replica: edde42ac7f914ad0806c9bbab60269e0, State: Running, Role: LEADER
I20260924 00:50:55.870724 10364 sys_catalog.cc:564] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:55.870682 10369 consensus_queue.cc:237] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde42ac7f914ad0806c9bbab60269e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40425 } }
I20260924 00:50:55.873472 10371 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edde42ac7f914ad0806c9bbab60269e0. Latest consensus state: current_term: 1 leader_uuid: "edde42ac7f914ad0806c9bbab60269e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde42ac7f914ad0806c9bbab60269e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40425 } } }
I20260924 00:50:55.873886 10371 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:55.874367 10370 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edde42ac7f914ad0806c9bbab60269e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edde42ac7f914ad0806c9bbab60269e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 40425 } } }
I20260924 00:50:55.874704 10370 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0 [sys.catalog]: This master's current role is: LEADER
W20260924 00:50:55.880676 10384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:50:55.880821 10384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:50:55.880968 10379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:55.882601 10379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:55.891578 10379 catalog_manager.cc:1347] Generated new cluster ID: d2d357a9706c4ad8b247cc33aa77c49e
I20260924 00:50:55.891693 10379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:55.909722 10379 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:55.911110 10379 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:55.928565 10379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edde42ac7f914ad0806c9bbab60269e0: Generated new TSK 0
I20260924 00:50:55.929520 10379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:50:55.942794 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40425
--builtin_ntp_servers=127.0.222.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 00:50:56.278666 10388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:56.278961 10388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:56.279071 10388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:56.286517 10388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:56.286636 10388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:56.286779 10388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:56.296200 10388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:56.297593 10388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:56.299461 10388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:56.312888 10393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:56.312973 10396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:56.312922 10394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:56.313907 10388 server_base.cc:1034] running on GCE node
I20260924 00:50:56.314574 10388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:56.315580 10388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:56.316892 10388 hybrid_clock.cc:648] HybridClock initialized: now 1790211056316836 us; error 43 us; skew 500 ppm
I20260924 00:50:56.317301 10388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:56.319664 10388 webserver.cc:460] Webserver started at http://127.0.222.1:44329/ using document root <none> and password file <none>
I20260924 00:50:56.320369 10388 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:56.320462 10388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:56.320783 10388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:56.323693 10388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "d1afcaeb5f0c415fb3dac033bdda2f9e"
format_stamp: "Formatted at 2026-09-24 00:50:56 on dist-test-slave-rtvz"
server_key: "7e4b3903b5bd403627bf14e04d5d5e4e"
server_key_iv: "83ec1c0ea2cf7a098f8d83213c07b149"
server_key_version: "encryptionkey@0"
I20260924 00:50:56.324501 10388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "d1afcaeb5f0c415fb3dac033bdda2f9e"
format_stamp: "Formatted at 2026-09-24 00:50:56 on dist-test-slave-rtvz"
server_key: "7e4b3903b5bd403627bf14e04d5d5e4e"
server_key_iv: "83ec1c0ea2cf7a098f8d83213c07b149"
server_key_version: "encryptionkey@0"
I20260924 00:50:56.329725 10388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 00:50:56.333176 10402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:56.334537 10388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:50:56.334698 10388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "d1afcaeb5f0c415fb3dac033bdda2f9e"
format_stamp: "Formatted at 2026-09-24 00:50:56 on dist-test-slave-rtvz"
server_key: "7e4b3903b5bd403627bf14e04d5d5e4e"
server_key_iv: "83ec1c0ea2cf7a098f8d83213c07b149"
server_key_version: "encryptionkey@0"
I20260924 00:50:56.334889 10388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:56.355039 10388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:56.355950 10388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:56.356251 10388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:56.357282 10388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:56.358784 10388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:56.358871 10388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:56.358973 10388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:56.359045 10388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:56.389958 10388 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:35897
I20260924 00:50:56.390030 10515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:35897 every 8 connection(s)
I20260924 00:50:56.391268 10388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:56.394469 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 10388
I20260924 00:50:56.394675 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:50:56.395546 888 external_mini_cluster.cc:1428] Setting key 546113299f976a1c0d953eca67777464
I20260924 00:50:56.398864 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40425
--builtin_ntp_servers=127.0.222.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 00:50:56.405294 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:56.393817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:49631 (local address 127.0.222.62:40425)
0924 00:50:56.394263 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:56.394268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:56.395849 (+ 1581us) server_negotiation.cc:407] Connection header received
0924 00:50:56.398344 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:56.398350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:56.398443 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:56.398557 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:56.399811 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:56.400590 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:56.402316 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:56.402777 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:56.402914 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:56.404420 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:56.404425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:56.404429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:56.404450 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:56.404498 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:56.404510 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:56.404586 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:56.404817 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 00:50:56.404920 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260924 00:50:56.406435 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:56.394543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:40425 (local address 127.0.222.1:49631)
0924 00:50:56.395657 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:56.395717 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:56.398052 (+ 2335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:56.398816 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:56.398826 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:56.398927 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:56.399616 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:56.399637 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:56.400747 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:56.400752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:56.402161 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:56.402175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:56.403029 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:56.403048 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:56.403305 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:56.404054 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:56.404057 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:56.404107 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:56.404221 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:56.404613 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:56.404624 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:56.404630 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:50:56.404792 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 00:50:56.405246 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":341,"threads_started":1}
I20260924 00:50:56.408140 10516 heartbeater.cc:347] Connected to a master server at 127.0.222.62:40425
I20260924 00:50:56.408440 10516 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:56.409305 10516 heartbeater.cc:511] Master 127.0.222.62:40425 requested a full tablet report, sending...
I20260924 00:50:56.411763 10328 ts_manager.cc:194] Registered new tserver with Master: d1afcaeb5f0c415fb3dac033bdda2f9e (127.0.222.1:35897)
I20260924 00:50:56.414034 10328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.1:49631
W20260924 00:50:56.732400 10520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:56.732709 10520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:56.732793 10520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:56.740603 10520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:56.740733 10520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:56.740844 10520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:56.750396 10520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:56.751662 10520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:56.753484 10520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:56.766841 10526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:56.766950 10528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:56.766976 10525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:56.768416 10520 server_base.cc:1034] running on GCE node
I20260924 00:50:56.769064 10520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:56.770126 10520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:56.771332 10520 hybrid_clock.cc:648] HybridClock initialized: now 1790211056771281 us; error 66 us; skew 500 ppm
I20260924 00:50:56.771685 10520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:56.774097 10520 webserver.cc:460] Webserver started at http://127.0.222.2:35063/ using document root <none> and password file <none>
I20260924 00:50:56.774803 10520 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:56.774906 10520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:56.775215 10520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:56.778158 10520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "666ac679f16949deaadc70f896d3903d"
format_stamp: "Formatted at 2026-09-24 00:50:56 on dist-test-slave-rtvz"
server_key: "5d1fc3f98db28600572044c87c9ad2f6"
server_key_iv: "2cb07373233aee3b465db2859a2fde9a"
server_key_version: "encryptionkey@0"
I20260924 00:50:56.778927 10520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "666ac679f16949deaadc70f896d3903d"
format_stamp: "Formatted at 2026-09-24 00:50:56 on dist-test-slave-rtvz"
server_key: "5d1fc3f98db28600572044c87c9ad2f6"
server_key_iv: "2cb07373233aee3b465db2859a2fde9a"
server_key_version: "encryptionkey@0"
I20260924 00:50:56.784324 10520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:50:56.788126 10534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:56.789732 10520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:56.789899 10520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "666ac679f16949deaadc70f896d3903d"
format_stamp: "Formatted at 2026-09-24 00:50:56 on dist-test-slave-rtvz"
server_key: "5d1fc3f98db28600572044c87c9ad2f6"
server_key_iv: "2cb07373233aee3b465db2859a2fde9a"
server_key_version: "encryptionkey@0"
I20260924 00:50:56.790122 10520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:56.806092 10520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:56.807001 10520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:56.807271 10520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:56.811573 10520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:56.813587 10520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:56.813711 10520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:56.813828 10520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:56.813902 10520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:56.855569 10520 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:33501
I20260924 00:50:56.855648 10647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:33501 every 8 connection(s)
I20260924 00:50:56.856878 10520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:56.862265 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 10520
I20260924 00:50:56.862419 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:50:56.862879 888 external_mini_cluster.cc:1428] Setting key 7735e9d3a798ac2a7d0a6ee256b0f8dc
I20260924 00:50:56.865474 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40425
--builtin_ntp_servers=127.0.222.20:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 00:50:56.869017 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:56.859368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:48675 (local address 127.0.222.62:40425)
0924 00:50:56.859581 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:56.859588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:56.860927 (+ 1339us) server_negotiation.cc:407] Connection header received
0924 00:50:56.862688 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:56.862694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:56.862782 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:56.862895 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:56.864026 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:56.864757 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:56.866418 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:56.866808 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:56.866917 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:56.868248 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:56.868253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:56.868256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:56.868274 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:56.868307 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:56.868318 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:56.868390 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:56.868605 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 00:50:56.868707 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 00:50:56.870118 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:56.859833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:40425 (local address 127.0.222.2:48675)
0924 00:50:56.860784 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:56.860823 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:56.862417 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:56.863103 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:56.863112 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:56.863210 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:56.863854 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:56.863873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:56.864899 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:56.864903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:56.866273 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:56.866286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:56.867018 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:56.867026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:56.867255 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:56.867936 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:56.867939 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:56.867979 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:56.868062 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:56.868403 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:56.868413 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:56.868421 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 00:50:56.868575 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 00:50:56.868985 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":279,"threads_started":1}
I20260924 00:50:56.871618 10648 heartbeater.cc:347] Connected to a master server at 127.0.222.62:40425
I20260924 00:50:56.871930 10648 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:56.872737 10648 heartbeater.cc:511] Master 127.0.222.62:40425 requested a full tablet report, sending...
I20260924 00:50:56.874320 10328 ts_manager.cc:194] Registered new tserver with Master: 666ac679f16949deaadc70f896d3903d (127.0.222.2:33501)
I20260924 00:50:56.875118 10328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.2:48675
W20260924 00:50:57.192798 10651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:57.193096 10651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:57.193228 10651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:57.200667 10651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:57.200785 10651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:57.200925 10651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:57.210085 10651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:40425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:57.211409 10651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:57.213317 10651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:57.226769 10657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:57.226704 10658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:57.226740 10660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:57.227444 10651 server_base.cc:1034] running on GCE node
I20260924 00:50:57.228055 10651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:57.228994 10651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:57.230217 10651 hybrid_clock.cc:648] HybridClock initialized: now 1790211057230147 us; error 63 us; skew 500 ppm
I20260924 00:50:57.230553 10651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:57.232748 10651 webserver.cc:460] Webserver started at http://127.0.222.3:42817/ using document root <none> and password file <none>
I20260924 00:50:57.233486 10651 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:57.233583 10651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:57.233899 10651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:57.236861 10651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "229a0d60e957427a8b0f5d3e89b9fd80"
format_stamp: "Formatted at 2026-09-24 00:50:57 on dist-test-slave-rtvz"
server_key: "52b7baa7aeea78cdca2225bc1a387a28"
server_key_iv: "ddfd18a3e4693ca7aee1fd718e1efe2d"
server_key_version: "encryptionkey@0"
I20260924 00:50:57.237761 10651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "229a0d60e957427a8b0f5d3e89b9fd80"
format_stamp: "Formatted at 2026-09-24 00:50:57 on dist-test-slave-rtvz"
server_key: "52b7baa7aeea78cdca2225bc1a387a28"
server_key_iv: "ddfd18a3e4693ca7aee1fd718e1efe2d"
server_key_version: "encryptionkey@0"
I20260924 00:50:57.243446 10651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:50:57.247099 10666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:57.248518 10651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:57.248673 10651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "229a0d60e957427a8b0f5d3e89b9fd80"
format_stamp: "Formatted at 2026-09-24 00:50:57 on dist-test-slave-rtvz"
server_key: "52b7baa7aeea78cdca2225bc1a387a28"
server_key_iv: "ddfd18a3e4693ca7aee1fd718e1efe2d"
server_key_version: "encryptionkey@0"
I20260924 00:50:57.248888 10651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:57.266073 10651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:57.267023 10651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:57.267304 10651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:57.268311 10651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:57.270030 10651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:57.270156 10651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:57.270282 10651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:57.270356 10651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:57.302074 10651 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:46753
I20260924 00:50:57.302109 10779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:46753 every 8 connection(s)
I20260924 00:50:57.303412 10651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:57.308178 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 10651
I20260924 00:50:57.308385 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:50:57.309012 888 external_mini_cluster.cc:1428] Setting key 789d908d84c052e7e0080f9630125002
I20260924 00:50:57.316159 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:57.305966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:33037 (local address 127.0.222.62:40425)
0924 00:50:57.306164 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:57.306170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:57.307596 (+ 1426us) server_negotiation.cc:407] Connection header received
0924 00:50:57.309895 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:57.309899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:57.309972 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:57.310049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:57.311333 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.312292 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.313836 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.314244 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.314336 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:57.315512 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:57.315517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:57.315520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:57.315539 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:57.315571 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:57.315583 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:57.315659 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:57.315815 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 00:50:57.315893 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:50:57.317137 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:57.306500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:40425 (local address 127.0.222.3:33037)
0924 00:50:57.307435 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:57.307477 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:57.309642 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:57.310276 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:57.310287 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:57.310401 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:57.311167 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:57.311192 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.312420 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.312424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:57.313711 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:50:57.313722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.314410 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.314414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:50:57.314608 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:57.315209 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:57.315211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:57.315258 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:57.315333 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:57.315653 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:57.315659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:57.315663 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:50:57.315801 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 00:50:57.316093 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":296,"threads_started":1}
I20260924 00:50:57.318462 10780 heartbeater.cc:347] Connected to a master server at 127.0.222.62:40425
I20260924 00:50:57.318821 10780 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:57.319717 10780 heartbeater.cc:511] Master 127.0.222.62:40425 requested a full tablet report, sending...
I20260924 00:50:57.321326 10328 ts_manager.cc:194] Registered new tserver with Master: 229a0d60e957427a8b0f5d3e89b9fd80 (127.0.222.3:46753)
I20260924 00:50:57.322105 10328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.3:33037
I20260924 00:50:57.324404 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:50:57.330349 10518 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:57.328974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.0.222.62:40425)
0924 00:50:57.329121 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:57.329125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:57.329566 (+ 441us) server_negotiation.cc:407] Connection header received
0924 00:50:57.329698 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:57.329702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:57.329776 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:57.329854 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:57.330169 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42744: BlockingRecv error: recv got EOF from 127.0.0.1:42744 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:50:57.330992 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 10388
I20260924 00:50:57.338176 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:57.331902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.0.222.1:35897)
0924 00:50:57.332440 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:57.332446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:57.332471 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:50:57.332524 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:57.332531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:57.332741 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:57.332856 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:57.333166 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.334229 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.335385 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.335918 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.336017 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:57.336873 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:57.336882 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:57.336894 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:57.337510 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:57.337558 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:57.337579 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:57.337671 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:57.337789 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 00:50:57.337898 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
I20260924 00:50:57.339433 10511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:57.420272 10516 heartbeater.cc:503] Master 127.0.222.62:40425 was elected leader, sending a full tablet report...
I20260924 00:50:57.422163 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 10388
I20260924 00:50:57.435398 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 10520
I20260924 00:50:57.442889 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:57.436331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.0.222.2:33501)
0924 00:50:57.436866 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:57.436873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:57.436899 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:50:57.436966 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:57.436972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:57.437187 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:57.437362 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:57.437722 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.438715 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.439879 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.440424 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.440513 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:57.441546 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:57.441553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:57.441567 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:57.442135 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:57.442185 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:57.442203 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:57.442306 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:57.442419 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 00:50:57.442546 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":280,"threads_started":1}
I20260924 00:50:57.444139 10643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:57.527127 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 10520
I20260924 00:50:57.542181 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 10651
I20260924 00:50:57.550400 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:57.543151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.222.3:46753)
0924 00:50:57.543763 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:57.543771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:57.543803 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:50:57.543875 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:57.543882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:57.544103 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:57.544248 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:57.544740 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.545876 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.547137 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:50:57.547777 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:50:57.547878 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:50:57.548867 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:57.548875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:57.548889 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:57.549545 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:57.549601 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:57.549622 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:57.549714 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:57.549849 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 00:50:57.549980 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":292,"threads_started":1}
I20260924 00:50:57.551684 10775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:57.632941 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 10651
I20260924 00:50:57.645293 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 10297
I20260924 00:50:57.646548 10358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:57.750080 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 10297
2026-09-24T00:50:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2412 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T00:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:50:57Z Disabled control of system clock
I20260924 00:50:57.815944 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36711
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:36711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:50:58.148077 10799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:58.148391 10799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:58.148519 10799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:58.155989 10799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:50:58.156100 10799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:58.156147 10799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:58.156225 10799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:50:58.156277 10799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:50:58.166313 10799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:36711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:58.167691 10799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:58.169476 10799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:58.181659 10805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:58.181732 10807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:58.181578 10804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:58.183059 10799 server_base.cc:1034] running on GCE node
I20260924 00:50:58.184005 10799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:58.185662 10799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:58.186902 10799 hybrid_clock.cc:648] HybridClock initialized: now 1790211058186851 us; error 42 us; skew 500 ppm
I20260924 00:50:58.187292 10799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:58.189664 10799 webserver.cc:460] Webserver started at http://127.0.222.62:36549/ using document root <none> and password file <none>
I20260924 00:50:58.190367 10799 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:58.190467 10799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:58.190766 10799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:58.193661 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "c9d5892e348f45b987643454b8ec6290"
format_stamp: "Formatted at 2026-09-24 00:50:58 on dist-test-slave-rtvz"
server_key: "335a323e419b908caa4289ff44b73cd0"
server_key_iv: "4dbf0bdf2a01360c0dbfea1c372e8da9"
server_key_version: "encryptionkey@0"
I20260924 00:50:58.194427 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "c9d5892e348f45b987643454b8ec6290"
format_stamp: "Formatted at 2026-09-24 00:50:58 on dist-test-slave-rtvz"
server_key: "335a323e419b908caa4289ff44b73cd0"
server_key_iv: "4dbf0bdf2a01360c0dbfea1c372e8da9"
server_key_version: "encryptionkey@0"
I20260924 00:50:58.199708 10799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:50:58.203088 10814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:58.204455 10799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:50:58.204614 10799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "c9d5892e348f45b987643454b8ec6290"
format_stamp: "Formatted at 2026-09-24 00:50:58 on dist-test-slave-rtvz"
server_key: "335a323e419b908caa4289ff44b73cd0"
server_key_iv: "4dbf0bdf2a01360c0dbfea1c372e8da9"
server_key_version: "encryptionkey@0"
I20260924 00:50:58.204813 10799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:58.210409 10799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:58.211127 10799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:58.211395 10799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:58.231645 10799 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:36711
I20260924 00:50:58.231663 10866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:36711 every 8 connection(s)
I20260924 00:50:58.233109 10799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:50:58.236961 10867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:50:58.237735 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 10799
I20260924 00:50:58.237915 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:50:58.238600 888 external_mini_cluster.cc:1428] Setting key 197018146bb1baa68068a3d56e9d16fa
I20260924 00:50:58.247772 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:58.240737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.0.222.62:36711)
0924 00:50:58.242098 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:58.242120 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:58.242179 (+ 59us) server_negotiation.cc:407] Connection header received
0924 00:50:58.244007 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:58.244035 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:58.244518 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:58.244890 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:58.246063 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:58.246070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:58.246084 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:58.246100 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:58.246165 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:58.246181 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:58.246357 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:58.246516 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 00:50:58.246786 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":456,"threads_started":1}
I20260924 00:50:58.251349 10867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290: Bootstrap starting.
I20260924 00:50:58.254444 10867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290: Neither blocks nor log segments found. Creating new log.
I20260924 00:50:58.255477 10867 log.cc:824] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290: Log is configured to *not* fsync() on all Append() calls
I20260924 00:50:58.258610 10867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290: No bootstrap required, opened a new log
I20260924 00:50:58.266533 10867 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5892e348f45b987643454b8ec6290" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36711 } }
I20260924 00:50:58.266957 10867 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:50:58.267069 10867 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d5892e348f45b987643454b8ec6290, State: Initialized, Role: FOLLOWER
I20260924 00:50:58.267642 10867 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5892e348f45b987643454b8ec6290" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36711 } }
I20260924 00:50:58.267874 10867 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:50:58.267997 10867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:50:58.268151 10867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:50:58.270359 10867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5892e348f45b987643454b8ec6290" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36711 } }
I20260924 00:50:58.270861 10867 leader_election.cc:302] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d5892e348f45b987643454b8ec6290; no voters:
I20260924 00:50:58.271433 10867 leader_election.cc:288] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:50:58.271570 10872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:50:58.272567 10872 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [term 1 LEADER]: Becoming Leader. State: Replica: c9d5892e348f45b987643454b8ec6290, State: Running, Role: LEADER
I20260924 00:50:58.273171 10872 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5892e348f45b987643454b8ec6290" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36711 } }
I20260924 00:50:58.273602 10867 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:50:58.275944 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d5892e348f45b987643454b8ec6290. Latest consensus state: current_term: 1 leader_uuid: "c9d5892e348f45b987643454b8ec6290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5892e348f45b987643454b8ec6290" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36711 } } }
I20260924 00:50:58.275910 10873 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d5892e348f45b987643454b8ec6290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d5892e348f45b987643454b8ec6290" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36711 } } }
I20260924 00:50:58.276315 10873 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:58.276319 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290 [sys.catalog]: This master's current role is: LEADER
I20260924 00:50:58.277136 10878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:50:58.282915 10878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:50:58.292339 10878 catalog_manager.cc:1347] Generated new cluster ID: dfce840825404c8bb039879be00da393
I20260924 00:50:58.292469 10878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:50:58.300639 10878 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:50:58.301065 10878 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:50:58.312060 10878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9d5892e348f45b987643454b8ec6290: Generated new TSK 0
I20260924 00:50:58.312934 10878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:50:58.324505 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36711
--builtin_ntp_servers=127.0.222.20:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:50:58.650844 10891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:58.651131 10891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:58.651247 10891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:58.658694 10891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:58.658818 10891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:58.658967 10891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:50:58.668522 10891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:58.669989 10891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:58.671806 10891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:58.684908 10897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:58.684877 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:58.684976 10898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:58.685652 10891 server_base.cc:1034] running on GCE node
I20260924 00:50:58.686295 10891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:58.687247 10891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:58.688463 10891 hybrid_clock.cc:648] HybridClock initialized: now 1790211058688406 us; error 48 us; skew 500 ppm
I20260924 00:50:58.688800 10891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:58.691141 10891 webserver.cc:460] Webserver started at http://127.0.222.1:39739/ using document root <none> and password file <none>
I20260924 00:50:58.691809 10891 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:58.691910 10891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:58.692224 10891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:58.696619 10891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "9b57b02ae1704bd3bffdbe0463eceb41"
format_stamp: "Formatted at 2026-09-24 00:50:58 on dist-test-slave-rtvz"
server_key: "4c433cbd658f78e3ffee3edd8870a8b0"
server_key_iv: "39f9e9654eae1b66495bb8844be4b3b1"
server_key_version: "encryptionkey@0"
I20260924 00:50:58.697464 10891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "9b57b02ae1704bd3bffdbe0463eceb41"
format_stamp: "Formatted at 2026-09-24 00:50:58 on dist-test-slave-rtvz"
server_key: "4c433cbd658f78e3ffee3edd8870a8b0"
server_key_iv: "39f9e9654eae1b66495bb8844be4b3b1"
server_key_version: "encryptionkey@0"
I20260924 00:50:58.702718 10891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:50:58.706313 10906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:58.707769 10891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:50:58.707962 10891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "9b57b02ae1704bd3bffdbe0463eceb41"
format_stamp: "Formatted at 2026-09-24 00:50:58 on dist-test-slave-rtvz"
server_key: "4c433cbd658f78e3ffee3edd8870a8b0"
server_key_iv: "39f9e9654eae1b66495bb8844be4b3b1"
server_key_version: "encryptionkey@0"
I20260924 00:50:58.708218 10891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:58.713976 10891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:58.714787 10891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:58.715020 10891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:58.715921 10891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:58.717377 10891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:58.717463 10891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:58.717548 10891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:58.717631 10891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:58.748589 10891 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:36977
I20260924 00:50:58.748637 11020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:36977 every 8 connection(s)
I20260924 00:50:58.749967 10891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:50:58.755970 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 10891
I20260924 00:50:58.756181 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:50:58.756968 888 external_mini_cluster.cc:1428] Setting key 666916974fa552c9d5c414f7a25a829a
I20260924 00:50:58.758967 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:58.752840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:34867 (local address 127.0.222.62:36711)
0924 00:50:58.753388 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:58.753393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:58.754466 (+ 1073us) server_negotiation.cc:407] Connection header received
0924 00:50:58.756579 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:58.756584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:58.756646 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:58.756750 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:58.758268 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:58.758273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:58.758276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:58.758293 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:58.758335 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:58.758347 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:58.758439 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:58.758646 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 00:50:58.758695 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":289,"threads_started":1}
I20260924 00:50:58.760294 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:58.753342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36711 (local address 127.0.222.1:34867)
0924 00:50:58.754293 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:58.754331 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:58.756222 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:58.757049 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:58.757064 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:58.757235 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:58.757920 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:58.757923 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:58.757964 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:58.758067 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:58.758444 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:58.758455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:58.758463 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 00:50:58.758749 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 00:50:58.759040 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":287,"threads_started":1}
I20260924 00:50:58.760290 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36711
--builtin_ntp_servers=127.0.222.20:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 00:50:58.761864 11021 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36711
I20260924 00:50:58.762157 11021 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:58.762755 11021 heartbeater.cc:511] Master 127.0.222.62:36711 requested a full tablet report, sending...
I20260924 00:50:58.764835 10831 ts_manager.cc:194] Registered new tserver with Master: 9b57b02ae1704bd3bffdbe0463eceb41 (127.0.222.1:36977)
W20260924 00:50:59.092723 11025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:59.093055 11025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:59.093189 11025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:59.100674 11025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:59.100811 11025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:59.100960 11025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:50:59.110615 11025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:59.112193 11025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:59.114090 11025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:59.127548 11031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:59.127722 11030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:59.127607 11034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:59.128439 11025 server_base.cc:1034] running on GCE node
I20260924 00:50:59.129046 11025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:59.130125 11025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:59.131354 11025 hybrid_clock.cc:648] HybridClock initialized: now 1790211059131303 us; error 46 us; skew 500 ppm
I20260924 00:50:59.131686 11025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:59.134145 11025 webserver.cc:460] Webserver started at http://127.0.222.2:46843/ using document root <none> and password file <none>
I20260924 00:50:59.134850 11025 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:59.134955 11025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:59.135258 11025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:59.138239 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "bf347189275348e5875ab47010a38240"
format_stamp: "Formatted at 2026-09-24 00:50:59 on dist-test-slave-rtvz"
server_key: "cf7e0af9cfc64bdd098d7bf558f0851f"
server_key_iv: "1aa991f0658e20fb205c061871cec860"
server_key_version: "encryptionkey@0"
I20260924 00:50:59.138995 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "bf347189275348e5875ab47010a38240"
format_stamp: "Formatted at 2026-09-24 00:50:59 on dist-test-slave-rtvz"
server_key: "cf7e0af9cfc64bdd098d7bf558f0851f"
server_key_iv: "1aa991f0658e20fb205c061871cec860"
server_key_version: "encryptionkey@0"
I20260924 00:50:59.144300 11025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:50:59.147758 11040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:59.149308 11025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:50:59.149474 11025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "bf347189275348e5875ab47010a38240"
format_stamp: "Formatted at 2026-09-24 00:50:59 on dist-test-slave-rtvz"
server_key: "cf7e0af9cfc64bdd098d7bf558f0851f"
server_key_iv: "1aa991f0658e20fb205c061871cec860"
server_key_version: "encryptionkey@0"
I20260924 00:50:59.149770 11025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:59.155242 11025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:59.155998 11025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:59.156237 11025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:59.157104 11025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:59.158541 11025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:59.158640 11025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:59.158749 11025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:59.158818 11025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:59.189695 11025 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:34037
I20260924 00:50:59.189766 11153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:34037 every 8 connection(s)
I20260924 00:50:59.191011 11025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:50:59.193344 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 11025
I20260924 00:50:59.193522 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:50:59.194006 888 external_mini_cluster.cc:1428] Setting key e55420d3e5ec61f723a751df72daaf35
I20260924 00:50:59.196846 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36711
--builtin_ntp_servers=127.0.222.20:39525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 00:50:59.199586 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:59.193630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:36149 (local address 127.0.222.62:36711)
0924 00:50:59.193836 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:59.193843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:59.195506 (+ 1663us) server_negotiation.cc:407] Connection header received
0924 00:50:59.197389 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:59.197394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:59.197457 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:59.197559 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:59.198826 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:59.198831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:59.198835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:59.198853 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:59.198890 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:59.198903 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:59.199012 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:59.199253 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 00:50:59.199314 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:50:59.200596 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:59.194317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36711 (local address 127.0.222.2:36149)
0924 00:50:59.195342 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:59.195387 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:59.197094 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:59.197751 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:59.197759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:59.197860 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:59.198522 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:59.198524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:59.198566 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:59.198657 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:59.199029 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:59.199039 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:59.199046 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:50:59.199314 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 00:50:59.199548 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
I20260924 00:50:59.202131 11154 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36711
I20260924 00:50:59.202445 11154 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:59.203074 11154 heartbeater.cc:511] Master 127.0.222.62:36711 requested a full tablet report, sending...
I20260924 00:50:59.204648 10831 ts_manager.cc:194] Registered new tserver with Master: bf347189275348e5875ab47010a38240 (127.0.222.2:34037)
W20260924 00:50:59.524963 11157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:50:59.525305 11157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:50:59.525447 11157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:50:59.532790 11157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:50:59.532899 11157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:50:59.533041 11157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:50:59.542447 11157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:39525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:50:59.543877 11157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:50:59.545749 11157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:50:59.558920 11162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:59.558827 11163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:50:59.558845 11165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:50:59.559684 11157 server_base.cc:1034] running on GCE node
I20260924 00:50:59.560318 11157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:50:59.561305 11157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:50:59.562512 11157 hybrid_clock.cc:648] HybridClock initialized: now 1790211059562455 us; error 49 us; skew 500 ppm
I20260924 00:50:59.562835 11157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:50:59.565266 11157 webserver.cc:460] Webserver started at http://127.0.222.3:42553/ using document root <none> and password file <none>
I20260924 00:50:59.565991 11157 fs_manager.cc:359] Metadata directory not provided
I20260924 00:50:59.566123 11157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:50:59.566473 11157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:50:59.569610 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "d9a687a4456e4933a847d711eda7aba4"
format_stamp: "Formatted at 2026-09-24 00:50:59 on dist-test-slave-rtvz"
server_key: "9ad1645ae10ce497d40bc60d224446eb"
server_key_iv: "3e74a793086caf8ed40453edefe35c9f"
server_key_version: "encryptionkey@0"
I20260924 00:50:59.570470 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "d9a687a4456e4933a847d711eda7aba4"
format_stamp: "Formatted at 2026-09-24 00:50:59 on dist-test-slave-rtvz"
server_key: "9ad1645ae10ce497d40bc60d224446eb"
server_key_iv: "3e74a793086caf8ed40453edefe35c9f"
server_key_version: "encryptionkey@0"
I20260924 00:50:59.576165 11157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 00:50:59.579782 11171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:59.581312 11157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 00:50:59.581491 11157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "d9a687a4456e4933a847d711eda7aba4"
format_stamp: "Formatted at 2026-09-24 00:50:59 on dist-test-slave-rtvz"
server_key: "9ad1645ae10ce497d40bc60d224446eb"
server_key_iv: "3e74a793086caf8ed40453edefe35c9f"
server_key_version: "encryptionkey@0"
I20260924 00:50:59.581758 11157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:50:59.587642 11157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:50:59.588471 11157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:50:59.588744 11157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:50:59.589746 11157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:50:59.591289 11157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:50:59.591403 11157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:59.591530 11157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:50:59.591609 11157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:50:59.624874 11157 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:42151
I20260924 00:50:59.624931 11284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:42151 every 8 connection(s)
I20260924 00:50:59.626427 11157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:50:59.628638 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 11157
I20260924 00:50:59.628795 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:50:59.629312 888 external_mini_cluster.cc:1428] Setting key b0fb4e70cb26cebdfe21ec27086e6cc1
I20260924 00:50:59.634984 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:59.629160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:55145 (local address 127.0.222.62:36711)
0924 00:50:59.629421 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:59.629428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:59.631041 (+ 1613us) server_negotiation.cc:407] Connection header received
0924 00:50:59.632905 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:59.632909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:59.632965 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:59.633088 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:59.634319 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:59.634324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:59.634327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:59.634343 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:59.634376 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:59.634387 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:59.634494 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:59.634711 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 00:50:59.634768 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 00:50:59.635865 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:59.629791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36711 (local address 127.0.222.3:55145)
0924 00:50:59.630851 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0924 00:50:59.630907 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 00:50:59.632661 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:50:59.633326 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:50:59.633336 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:50:59.633432 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:50:59.634018 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:50:59.634020 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:50:59.634057 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:50:59.634152 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:50:59.634467 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:50:59.634473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:50:59.634478 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:50:59.634720 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 00:50:59.634927 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":312,"threads_started":1}
I20260924 00:50:59.637326 11285 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36711
I20260924 00:50:59.637615 11285 heartbeater.cc:464] Registering TS with master...
I20260924 00:50:59.638212 11285 heartbeater.cc:511] Master 127.0.222.62:36711 requested a full tablet report, sending...
I20260924 00:50:59.639635 10831 ts_manager.cc:194] Registered new tserver with Master: d9a687a4456e4933a847d711eda7aba4 (127.0.222.3:42151)
I20260924 00:50:59.645052 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:50:59.651196 11023 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:50:59.649445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.0.222.62:36711)
0924 00:50:59.649673 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:59.649680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:59.650285 (+ 605us) server_negotiation.cc:407] Connection header received
0924 00:50:59.650418 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:59.650422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:59.650482 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:59.650586 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:59.650902 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46724: BlockingRecv error: recv got EOF from 127.0.0.1:46724 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:50:59.651577 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 10891
I20260924 00:50:59.655933 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:59.652847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.0.222.1:36977)
0924 00:50:59.653501 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:59.653510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:59.653675 (+ 165us) server_negotiation.cc:407] Connection header received
0924 00:50:59.653852 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:59.653859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:59.654122 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:59.654269 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:59.655230 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:59.655236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:59.655249 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:59.655268 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:59.655318 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:59.655333 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:59.655451 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:59.655571 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 00:50:59.655673 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":332,"threads_started":1}
I20260924 00:50:59.657373 11016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:59.739183 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 10891
I20260924 00:50:59.753141 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 11025
I20260924 00:50:59.756587 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:59.754100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.0.222.2:34037)
0924 00:50:59.754683 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:59.754689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:59.754716 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:50:59.754778 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:59.754783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:59.754987 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:59.755116 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:59.755969 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:59.755975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:59.755987 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:59.756003 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:59.756042 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:59.756056 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:59.756158 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:59.756267 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 00:50:59.756348 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":282,"threads_started":1}
I20260924 00:50:59.757818 11149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:59.839077 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 11025
I20260924 00:50:59.851919 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 11157
I20260924 00:50:59.858457 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:50:59.852862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.0.222.3:42151)
0924 00:50:59.853504 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 00:50:59.853511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:50:59.853542 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:50:59.853613 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:50:59.853620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:50:59.853829 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:50:59.853949 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:50:59.857735 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:50:59.857742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:50:59.857757 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:50:59.857772 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:50:59.857828 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:50:59.857847 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:50:59.857983 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:50:59.858030 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 00:50:59.858132 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":283,"threads_started":1}
I20260924 00:50:59.859776 11280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:50:59.941097 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 11157
I20260924 00:50:59.953915 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 10799
I20260924 00:50:59.955240 10862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:00.058467 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 10799
2026-09-24T00:51:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2308 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:51:00 dist-test-slave-rtvz krb5kdc[11303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:51:00 dist-test-slave-rtvz krb5kdc[11303](info): set up 2 sockets
Sep 24 00:51:00 dist-test-slave-rtvz krb5kdc[11303](info): commencing operation
krb5kdc: starting...
W20260924 00:51:01.089187 888 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:51:01 dist-test-slave-rtvz krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:51:01.333245 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45533
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:51:01.659312 11316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:51:01.659637 11316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:51:01.659741 11316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:51:01.667261 11316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:51:01.667392 11316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:51:01.667439 11316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:51:01.667501 11316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:51:01.667552 11316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:51:01.678162 11316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:51:01.679610 11316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:51:01.680931 11316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:51:01.693082 11324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:01.693127 11321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:01.693162 11322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:01.694180 11316 server_base.cc:1034] running on GCE node
I20260924 00:51:01.695122 11316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:51:01.696861 11316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:51:01.698114 11316 hybrid_clock.cc:648] HybridClock initialized: now 1790211061698046 us; error 59 us; skew 500 ppm
I20260924 00:51:01.698540 11316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:51:01 dist-test-slave-rtvz krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:51:01.702077 11316 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:51:01.703771 11316 webserver.cc:460] Webserver started at http://127.0.222.62:34479/ using document root <none> and password file <none>
I20260924 00:51:01.704496 11316 fs_manager.cc:359] Metadata directory not provided
I20260924 00:51:01.704624 11316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:51:01.704942 11316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:51:01.708154 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "d5df9cd2135847ddaa33b7f862005042"
format_stamp: "Formatted at 2026-09-24 00:51:01 on dist-test-slave-rtvz"
server_key: "c4df55406efdc804cef94a8530a594d4"
server_key_iv: "a1cde3e179e1d263eca23a7d3e53e919"
server_key_version: "encryptionkey@0"
I20260924 00:51:01.708954 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "d5df9cd2135847ddaa33b7f862005042"
format_stamp: "Formatted at 2026-09-24 00:51:01 on dist-test-slave-rtvz"
server_key: "c4df55406efdc804cef94a8530a594d4"
server_key_iv: "a1cde3e179e1d263eca23a7d3e53e919"
server_key_version: "encryptionkey@0"
I20260924 00:51:01.714555 11316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 00:51:01.718539 11332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:01.720258 11316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:51:01.720464 11316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "d5df9cd2135847ddaa33b7f862005042"
format_stamp: "Formatted at 2026-09-24 00:51:01 on dist-test-slave-rtvz"
server_key: "c4df55406efdc804cef94a8530a594d4"
server_key_iv: "a1cde3e179e1d263eca23a7d3e53e919"
server_key_version: "encryptionkey@0"
I20260924 00:51:01.720678 11316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:51:01.763250 11316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:51:01.767612 11316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:51:01.767897 11316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:51:01.786362 11316 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45533
I20260924 00:51:01.786386 11383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45533 every 8 connection(s)
I20260924 00:51:01.787828 11316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:51:01.791777 11384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:51:01.795392 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 11316
I20260924 00:51:01.795614 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:51:01.796365 888 external_mini_cluster.cc:1428] Setting key eef57f6a44d7e22ee4d360af1a8fbefe
I20260924 00:51:01.803354 11384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042: Bootstrap starting.
I20260924 00:51:01.807163 11384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042: Neither blocks nor log segments found. Creating new log.
I20260924 00:51:01.808355 11384 log.cc:824] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042: Log is configured to *not* fsync() on all Append() calls
I20260924 00:51:01.811939 11384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042: No bootstrap required, opened a new log
Sep 24 00:51:01 dist-test-slave-rtvz krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:51:01.820123 11384 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5df9cd2135847ddaa33b7f862005042" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45533 } }
I20260924 00:51:01.820560 11384 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:51:01.820672 11384 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5df9cd2135847ddaa33b7f862005042, State: Initialized, Role: FOLLOWER
I20260924 00:51:01.820722 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:01.798579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42370 (local address 127.0.222.62:45533)
0924 00:51:01.799363 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:01.799381 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:01.799424 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:51:01.801722 (+ 2298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:01.801751 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:01.802187 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:01.802536 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:01.804406 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:01.805843 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:01.807606 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:01.808325 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:01.808458 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:01.813129 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:01.813138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:01.813154 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:01.813192 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:01.816340 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:01.817134 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:01.817141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:01.817148 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:01.817378 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:01.817766 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:01.817770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:01.817773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:01.818393 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:01.818715 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:01.819134 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 00:51:01.819585 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":330,"threads_started":1}
I20260924 00:51:01.821375 11384 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5df9cd2135847ddaa33b7f862005042" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45533 } }
I20260924 00:51:01.821617 11384 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:51:01.821748 11384 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:51:01.821921 11384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:51:01.824891 11384 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5df9cd2135847ddaa33b7f862005042" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45533 } }
I20260924 00:51:01.825435 11384 leader_election.cc:302] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5df9cd2135847ddaa33b7f862005042; no voters:
I20260924 00:51:01.826040 11384 leader_election.cc:288] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:51:01.826153 11389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:51:01.827409 11389 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [term 1 LEADER]: Becoming Leader. State: Replica: d5df9cd2135847ddaa33b7f862005042, State: Running, Role: LEADER
I20260924 00:51:01.827987 11389 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5df9cd2135847ddaa33b7f862005042" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45533 } }
I20260924 00:51:01.828392 11384 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:51:01.830730 11390 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5df9cd2135847ddaa33b7f862005042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5df9cd2135847ddaa33b7f862005042" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45533 } } }
I20260924 00:51:01.831132 11390 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [sys.catalog]: This master's current role is: LEADER
I20260924 00:51:01.831115 11391 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5df9cd2135847ddaa33b7f862005042. Latest consensus state: current_term: 1 leader_uuid: "d5df9cd2135847ddaa33b7f862005042" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5df9cd2135847ddaa33b7f862005042" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45533 } } }
I20260924 00:51:01.831432 11391 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042 [sys.catalog]: This master's current role is: LEADER
I20260924 00:51:01.833375 11394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:51:01.839381 11394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:51:01.850504 11394 catalog_manager.cc:1347] Generated new cluster ID: fd44293936474d36a52fab2ffba782d6
I20260924 00:51:01.850688 11394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:51:01.862409 11394 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:51:01.863843 11394 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:51:01.870307 11394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042: Generated new TSK 0
I20260924 00:51:01.871291 11394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:51:01.926060 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45533
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:51:02.274940 11410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:51:02.275238 11410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:51:02.275360 11410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:51:02.282981 11410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:51:02.283120 11410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:51:02.283283 11410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:51:02.293639 11410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:51:02.295030 11410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:51:02.296480 11410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:51:02.310425 11418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:02.310460 11415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:02.310482 11416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:02.311777 11410 server_base.cc:1034] running on GCE node
I20260924 00:51:02.312441 11410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:51:02.313549 11410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:51:02.314788 11410 hybrid_clock.cc:648] HybridClock initialized: now 1790211062314717 us; error 63 us; skew 500 ppm
I20260924 00:51:02.315131 11410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:51:02 dist-test-slave-rtvz krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:51:02.318475 11410 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:51:02.320144 11410 webserver.cc:460] Webserver started at http://127.0.222.1:33079/ using document root <none> and password file <none>
I20260924 00:51:02.320976 11410 fs_manager.cc:359] Metadata directory not provided
I20260924 00:51:02.321077 11410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:51:02.321389 11410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:51:02.324419 11410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "321f617457ef4f64bca6808ac677c660"
format_stamp: "Formatted at 2026-09-24 00:51:02 on dist-test-slave-rtvz"
server_key: "a2e42cab62c22cc35b20eb0f2ae7cb3f"
server_key_iv: "2d8ddc85ed851818375a275b960e449d"
server_key_version: "encryptionkey@0"
I20260924 00:51:02.325260 11410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "321f617457ef4f64bca6808ac677c660"
format_stamp: "Formatted at 2026-09-24 00:51:02 on dist-test-slave-rtvz"
server_key: "a2e42cab62c22cc35b20eb0f2ae7cb3f"
server_key_iv: "2d8ddc85ed851818375a275b960e449d"
server_key_version: "encryptionkey@0"
I20260924 00:51:02.330834 11410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 00:51:02.334805 11426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:02.336333 11410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:51:02.336498 11410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "321f617457ef4f64bca6808ac677c660"
format_stamp: "Formatted at 2026-09-24 00:51:02 on dist-test-slave-rtvz"
server_key: "a2e42cab62c22cc35b20eb0f2ae7cb3f"
server_key_iv: "2d8ddc85ed851818375a275b960e449d"
server_key_version: "encryptionkey@0"
I20260924 00:51:02.336689 11410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:51:02.356966 11410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:51:02.361335 11410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:51:02.361611 11410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:51:02.362597 11410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:51:02.364019 11410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:51:02.364089 11410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:02.364207 11410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:51:02.364264 11410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:02.395089 11410 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44623
I20260924 00:51:02.395149 11538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44623 every 8 connection(s)
I20260924 00:51:02.396355 11410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:51:02.399892 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 11410
I20260924 00:51:02.400072 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:51:02.400745 888 external_mini_cluster.cc:1428] Setting key 88ce068148e806e9710ac12500cde115
Sep 24 00:51:02 dist-test-slave-rtvz krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:51:02.416246 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:02.398890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:43787 (local address 127.0.222.62:45533)
0924 00:51:02.399299 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:02.399303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:02.400305 (+ 1002us) server_negotiation.cc:407] Connection header received
0924 00:51:02.402343 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:02.402348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:02.402438 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:02.402533 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:02.404050 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:02.404786 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:02.406432 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:02.406863 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:02.406979 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:02.410598 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:02.410604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:02.410607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:02.410640 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:02.413118 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:02.413928 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:02.413932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:02.413935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:02.414057 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:02.414482 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:02.414485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:02.414488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:02.414771 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:02.414901 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:02.415627 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 00:51:02.415886 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":217,"threads_started":1}
I20260924 00:51:02.417335 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:02.399372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45533 (local address 127.0.222.1:43787)
0924 00:51:02.400141 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:02.400191 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:02.402093 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:02.402777 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:02.402785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:02.403274 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:51:02.403893 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:02.403913 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:02.404925 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:02.404929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:02.406288 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:02.406300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:02.407102 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:02.407110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:02.407336 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:02.407970 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:51:02.407998 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:51:02.410365 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:51:02.413312 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:02.413323 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:02.413342 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:02.413796 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:02.414162 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:02.414167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:02.414170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:02.414369 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:02.414894 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:51:02.414902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:51:02.415261 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 00:51:02.415505 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 00:51:02.416069 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":304,"threads_started":1}
I20260924 00:51:02.418964 11539 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45533
I20260924 00:51:02.419318 11539 heartbeater.cc:464] Registering TS with master...
I20260924 00:51:02.420297 11539 heartbeater.cc:511] Master 127.0.222.62:45533 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:51:02.422710 11349 ts_manager.cc:194] Registered new tserver with Master: 321f617457ef4f64bca6808ac677c660 (127.0.222.1:44623)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260924 00:51:02.424283 11349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:43787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:51:02.451915 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45533
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:51:02.781169 11546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:51:02.781472 11546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:51:02.781569 11546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:51:02.789043 11546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:51:02.789181 11546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:51:02.789364 11546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:51:02.799572 11546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:51:02.800846 11546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:51:02.802222 11546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:51:02.815376 11554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:02.815413 11552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:02.815439 11551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:02.816260 11546 server_base.cc:1034] running on GCE node
I20260924 00:51:02.816871 11546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:51:02.817901 11546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:51:02.819160 11546 hybrid_clock.cc:648] HybridClock initialized: now 1790211062819098 us; error 47 us; skew 500 ppm
I20260924 00:51:02.819504 11546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:51:02 dist-test-slave-rtvz krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:51:02.823181 11546 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:51:02.825022 11546 webserver.cc:460] Webserver started at http://127.0.222.2:37035/ using document root <none> and password file <none>
I20260924 00:51:02.826047 11546 fs_manager.cc:359] Metadata directory not provided
I20260924 00:51:02.826197 11546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:51:02.826561 11546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:51:02.830735 11546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "1efa11d703d04c938fc24412a7347bba"
format_stamp: "Formatted at 2026-09-24 00:51:02 on dist-test-slave-rtvz"
server_key: "bdf1116d39bcea85dbd82355f66defd3"
server_key_iv: "4a6e968e2b927e32c8d4e38607e32ffc"
server_key_version: "encryptionkey@0"
I20260924 00:51:02.831832 11546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "1efa11d703d04c938fc24412a7347bba"
format_stamp: "Formatted at 2026-09-24 00:51:02 on dist-test-slave-rtvz"
server_key: "bdf1116d39bcea85dbd82355f66defd3"
server_key_iv: "4a6e968e2b927e32c8d4e38607e32ffc"
server_key_version: "encryptionkey@0"
I20260924 00:51:02.837847 11546 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 00:51:02.841642 11561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:02.843225 11546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:51:02.843372 11546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "1efa11d703d04c938fc24412a7347bba"
format_stamp: "Formatted at 2026-09-24 00:51:02 on dist-test-slave-rtvz"
server_key: "bdf1116d39bcea85dbd82355f66defd3"
server_key_iv: "4a6e968e2b927e32c8d4e38607e32ffc"
server_key_version: "encryptionkey@0"
I20260924 00:51:02.843568 11546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:51:02.859463 11546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:51:02.863927 11546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:51:02.864187 11546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:51:02.865130 11546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:51:02.866495 11546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:51:02.866573 11546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:02.866667 11546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:51:02.866719 11546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:02.897325 11546 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41175
I20260924 00:51:02.897400 11674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41175 every 8 connection(s)
I20260924 00:51:02.898633 11546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:51:02.904258 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 11546
I20260924 00:51:02.904430 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:51:02.904902 888 external_mini_cluster.cc:1428] Setting key 97db3b471396c0aff1f2097fdc47c5f9
Sep 24 00:51:02 dist-test-slave-rtvz krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:51:02.919204 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:02.901078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:46055 (local address 127.0.222.62:45533)
0924 00:51:02.901319 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:02.901324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:02.902465 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 00:51:02.904281 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:02.904288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:02.904375 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:02.904494 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:02.906209 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:02.907093 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:02.908796 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:02.909423 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:02.909542 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:02.913263 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:02.913284 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:02.913288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:02.913329 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:02.915891 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:02.916820 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:02.916825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:02.916827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:02.916952 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:02.917466 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:02.917470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:02.917472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:02.917773 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:02.917883 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:02.918766 (+ 883us) server_negotiation.cc:299] Negotiation successful
0924 00:51:02.918958 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 00:51:02.920536 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:02.901578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45533 (local address 127.0.222.2:46055)
0924 00:51:02.902310 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:02.902349 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:02.904012 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:02.904784 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:02.904793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:02.905390 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:51:02.906060 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:02.906079 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:02.907264 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:02.907269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:02.908630 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:02.908644 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:02.909717 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:02.909724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:02.909954 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:02.910647 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:51:02.910673 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:51:02.912959 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:51:02.916073 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:02.916085 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:02.916105 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:02.916695 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:02.917045 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:02.917050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:02.917061 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:02.917348 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:02.917883 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:51:02.917890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:51:02.918342 (+ 452us) client_negotiation.cc:769] Sending connection context
0924 00:51:02.918643 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 00:51:02.919196 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":292,"threads_started":1}
I20260924 00:51:02.922019 11675 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45533
I20260924 00:51:02.922325 11675 heartbeater.cc:464] Registering TS with master...
I20260924 00:51:02.923069 11675 heartbeater.cc:511] Master 127.0.222.62:45533 requested a full tablet report, sending...
I20260924 00:51:02.924548 11349 ts_manager.cc:194] Registered new tserver with Master: 1efa11d703d04c938fc24412a7347bba (127.0.222.2:41175)
I20260924 00:51:02.925473 11349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:46055
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:51:02.957831 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45533
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:51:03.282323 11680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:51:03.282624 11680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:51:03.282723 11680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:51:03.290006 11680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:51:03.290131 11680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:51:03.290252 11680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:51:03.300280 11680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:51:03.301680 11680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:51:03.303095 11680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:51:03.316349 11685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:03.316421 11686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:03.316449 11688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:03.317380 11680 server_base.cc:1034] running on GCE node
I20260924 00:51:03.318107 11680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:51:03.319070 11680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:51:03.320279 11680 hybrid_clock.cc:648] HybridClock initialized: now 1790211063320233 us; error 57 us; skew 500 ppm
I20260924 00:51:03.320593 11680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:51:03 dist-test-slave-rtvz krb5kdc[11303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:51:03.323771 11680 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:51:03.325314 11680 webserver.cc:460] Webserver started at http://127.0.222.3:33245/ using document root <none> and password file <none>
I20260924 00:51:03.326020 11680 fs_manager.cc:359] Metadata directory not provided
I20260924 00:51:03.326130 11680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:51:03.326416 11680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:51:03.329427 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "8e9d873796a2496ab0afae5438d5c425"
format_stamp: "Formatted at 2026-09-24 00:51:03 on dist-test-slave-rtvz"
server_key: "bcf01742906a43952ce93d10b27010b9"
server_key_iv: "b5d9a5ab16bd16af927810bb242ce4e1"
server_key_version: "encryptionkey@0"
I20260924 00:51:03.330210 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "8e9d873796a2496ab0afae5438d5c425"
format_stamp: "Formatted at 2026-09-24 00:51:03 on dist-test-slave-rtvz"
server_key: "bcf01742906a43952ce93d10b27010b9"
server_key_iv: "b5d9a5ab16bd16af927810bb242ce4e1"
server_key_version: "encryptionkey@0"
I20260924 00:51:03.335762 11680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:51:03.339744 11695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:03.341301 11680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:51:03.341463 11680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "8e9d873796a2496ab0afae5438d5c425"
format_stamp: "Formatted at 2026-09-24 00:51:03 on dist-test-slave-rtvz"
server_key: "bcf01742906a43952ce93d10b27010b9"
server_key_iv: "b5d9a5ab16bd16af927810bb242ce4e1"
server_key_version: "encryptionkey@0"
I20260924 00:51:03.341742 11680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:51:03.361819 11680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:51:03.366156 11680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:51:03.366438 11680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:51:03.367448 11680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:51:03.368922 11680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:51:03.369004 11680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:03.369125 11680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:51:03.369184 11680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:03.400527 11680 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:38177
I20260924 00:51:03.400576 11809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:38177 every 8 connection(s)
I20260924 00:51:03.401969 11680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:51:03.409488 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 11680
I20260924 00:51:03.409669 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:51:03.410125 888 external_mini_cluster.cc:1428] Setting key 96da3d68ba4069bf06c3173a985a3a93
Sep 24 00:51:03 dist-test-slave-rtvz krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:51:03.421525 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.404413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:47025 (local address 127.0.222.62:45533)
0924 00:51:03.404596 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.404600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.405875 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 00:51:03.407703 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.407708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.407823 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.407912 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:03.409584 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.410430 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.412159 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.412574 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.412685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.416163 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:03.416168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:03.416171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:03.416204 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:03.418647 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.419379 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.419383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.419385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.419508 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.419900 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.419904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.419906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.420165 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:03.420292 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.420956 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.421169 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:51:03.422463 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.404938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45533 (local address 127.0.222.3:47025)
0924 00:51:03.405694 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:03.405743 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:03.407428 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:03.408167 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:03.408176 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:03.408695 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:51:03.409391 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.409419 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.410603 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.410608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.412021 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.412033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.412785 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.412793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.413086 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.413714 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:51:03.413746 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:51:03.415953 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:51:03.418814 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:03.418823 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:03.418842 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:03.419259 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:03.419605 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:03.419610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:03.419614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:03.419790 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:03.420287 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:51:03.420294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:51:03.420603 (+ 309us) client_negotiation.cc:769] Sending connection context
0924 00:51:03.420830 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 00:51:03.421397 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":318,"threads_started":1}
I20260924 00:51:03.423887 11810 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45533
I20260924 00:51:03.424193 11810 heartbeater.cc:464] Registering TS with master...
I20260924 00:51:03.424923 11810 heartbeater.cc:511] Master 127.0.222.62:45533 requested a full tablet report, sending...
I20260924 00:51:03.426321 11349 ts_manager.cc:194] Registered new tserver with Master: 8e9d873796a2496ab0afae5438d5c425 (127.0.222.3:38177)
I20260924 00:51:03.427114 11349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:47025
I20260924 00:51:03.427342 11539 heartbeater.cc:503] Master 127.0.222.62:45533 was elected leader, sending a full tablet report...
I20260924 00:51:03.437856 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:51:03.454679 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.442257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.0.222.62:45533)
0924 00:51:03.442414 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.442421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.442928 (+ 507us) server_negotiation.cc:407] Connection header received
0924 00:51:03.443092 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.443096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.443160 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.443238 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:03.444600 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.445403 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.446637 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.447055 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.447157 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.449410 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:03.449415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:03.449418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:03.449454 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:03.451932 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.452603 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.452609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.452612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.452751 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.453108 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.453112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.453114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.453495 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:03.453748 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.454147 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.454373 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:51:03.458817 11349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:51:03.466020 11349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:51:03.488689 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.479047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41175 (local address 127.0.0.1:34346)
0924 00:51:03.479850 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:03.479888 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:03.480077 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:03.481307 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:03.481316 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:03.481350 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:03.481831 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.481850 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.484092 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.484097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.485727 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.485741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.487575 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.487581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.487860 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.487923 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 00:51:03.488077 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 00:51:03.488277 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":21,"thread_start_us":406,"threads_started":1}
I20260924 00:51:03.489596 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.479299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.222.2:41175)
0924 00:51:03.480642 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.480651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.480677 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:51:03.480777 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.480784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.481043 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.482060 (+ 1017us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:03.482676 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.483958 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.485898 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.487380 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.487510 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.487708 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.488962 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.489240 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180,"thread_start_us":1024,"threads_started":1}
I20260924 00:51:03.490001 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.480401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44623 (local address 127.0.0.1:35812)
0924 00:51:03.480943 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:03.480958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:03.481076 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:03.482762 (+ 1686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:03.482767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:03.482796 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:03.483226 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.483238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.485392 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.485398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.487388 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.487400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.489250 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.489256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.489456 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.489488 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:51:03.489566 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 00:51:03.489696 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":1,"thread_start_us":312,"threads_started":1}
I20260924 00:51:03.491108 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.481055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:38177 (local address 127.0.0.1:55244)
0924 00:51:03.481857 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:03.481874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:03.482001 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:03.483194 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:03.483199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:03.483226 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:03.483699 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.483711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.485714 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.485718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.487921 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.487932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.489409 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.489414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.489527 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.489554 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 00:51:03.489606 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:51:03.490713 (+ 1107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":74,"thread_start_us":502,"threads_started":1}
I20260924 00:51:03.491370 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.481545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.0.222.3:38177)
0924 00:51:03.482619 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.482626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.482661 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:51:03.482748 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.482755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.482968 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.483889 (+ 921us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:03.484154 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.485583 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.488068 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.489093 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.489947 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.490321 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.490384 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.490540 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":752,"threads_started":1}
I20260924 00:51:03.491875 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.480698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.0.222.1:44623)
0924 00:51:03.482166 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.482174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.482201 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:51:03.482296 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.482304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.482560 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.483418 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:03.483688 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.485245 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.487578 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.489007 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.490218 (+ 1211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.490440 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.490511 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.491642 (+ 1131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":1143,"threads_started":1}
I20260924 00:51:03.493837 11609 tablet_service.cc:1467] Processing CreateTablet for tablet 29294bd0a7344b7a97e94ce57d16b6f3 (DEFAULT_TABLE table=test-table [id=bff6cac420ab4a4d81b512d6b5fbdab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:51:03.494513 11744 tablet_service.cc:1467] Processing CreateTablet for tablet 29294bd0a7344b7a97e94ce57d16b6f3 (DEFAULT_TABLE table=test-table [id=bff6cac420ab4a4d81b512d6b5fbdab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:51:03.495366 11609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29294bd0a7344b7a97e94ce57d16b6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:51:03.495736 11744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29294bd0a7344b7a97e94ce57d16b6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:51:03.495249 11474 tablet_service.cc:1467] Processing CreateTablet for tablet 29294bd0a7344b7a97e94ce57d16b6f3 (DEFAULT_TABLE table=test-table [id=bff6cac420ab4a4d81b512d6b5fbdab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:51:03.496580 11474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29294bd0a7344b7a97e94ce57d16b6f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:51:03.507159 11828 tablet_bootstrap.cc:492] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: Bootstrap starting.
I20260924 00:51:03.511152 11830 tablet_bootstrap.cc:492] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: Bootstrap starting.
I20260924 00:51:03.511557 11828 tablet_bootstrap.cc:654] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: Neither blocks nor log segments found. Creating new log.
I20260924 00:51:03.512805 11828 log.cc:824] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: Log is configured to *not* fsync() on all Append() calls
I20260924 00:51:03.514340 11829 tablet_bootstrap.cc:492] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: Bootstrap starting.
I20260924 00:51:03.514634 11830 tablet_bootstrap.cc:654] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: Neither blocks nor log segments found. Creating new log.
I20260924 00:51:03.515861 11830 log.cc:824] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: Log is configured to *not* fsync() on all Append() calls
I20260924 00:51:03.515942 11828 tablet_bootstrap.cc:492] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: No bootstrap required, opened a new log
I20260924 00:51:03.516196 11828 ts_tablet_manager.cc:1397] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 00:51:03.517635 11829 tablet_bootstrap.cc:654] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: Neither blocks nor log segments found. Creating new log.
I20260924 00:51:03.518803 11829 log.cc:824] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: Log is configured to *not* fsync() on all Append() calls
I20260924 00:51:03.519011 11830 tablet_bootstrap.cc:492] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: No bootstrap required, opened a new log
I20260924 00:51:03.519270 11830 ts_tablet_manager.cc:1397] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 00:51:03.521960 11829 tablet_bootstrap.cc:492] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: No bootstrap required, opened a new log
I20260924 00:51:03.522246 11829 ts_tablet_manager.cc:1397] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 00:51:03.524684 11828 raft_consensus.cc:348] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.525072 11828 raft_consensus.cc:374] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:51:03.525166 11828 raft_consensus.cc:729] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9d873796a2496ab0afae5438d5c425, State: Initialized, Role: FOLLOWER
I20260924 00:51:03.525794 11828 consensus_queue.cc:260] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.526851 11810 heartbeater.cc:503] Master 127.0.222.62:45533 was elected leader, sending a full tablet report...
I20260924 00:51:03.527340 11828 ts_tablet_manager.cc:1428] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 00:51:03.529611 11830 raft_consensus.cc:348] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.530179 11830 raft_consensus.cc:374] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:51:03.530285 11830 raft_consensus.cc:729] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321f617457ef4f64bca6808ac677c660, State: Initialized, Role: FOLLOWER
I20260924 00:51:03.530953 11830 consensus_queue.cc:260] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.532749 11830 ts_tablet_manager.cc:1428] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 00:51:03.535225 11829 raft_consensus.cc:348] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.535564 11829 raft_consensus.cc:374] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:51:03.535665 11829 raft_consensus.cc:729] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1efa11d703d04c938fc24412a7347bba, State: Initialized, Role: FOLLOWER
I20260924 00:51:03.536307 11829 consensus_queue.cc:260] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.537457 11675 heartbeater.cc:503] Master 127.0.222.62:45533 was elected leader, sending a full tablet report...
I20260924 00:51:03.537961 11829 ts_tablet_manager.cc:1428] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260924 00:51:03.553561 11834 raft_consensus.cc:482] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:51:03.553905 11834 raft_consensus.cc:504] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.555629 11834 leader_election.cc:288] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1efa11d703d04c938fc24412a7347bba (127.0.222.2:41175), 321f617457ef4f64bca6808ac677c660 (127.0.222.1:44623)
I20260924 00:51:03.562013 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.556137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:46171 (local address 127.0.222.2:41175)
0924 00:51:03.556342 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.556347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.556364 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:51:03.556428 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.556432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.556486 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.556560 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:03.557009 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.557935 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.559927 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.561124 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.561294 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.561387 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.561721 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.561823 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 00:51:03.562124 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.555885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41175 (local address 127.0.222.3:46171)
0924 00:51:03.556082 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:03.556104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:03.556242 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:03.556591 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:03.556595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:03.556621 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:03.556842 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.556854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.558081 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.558085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.559771 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.559792 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.561302 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.561307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.561503 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.561538 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:51:03.561710 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 00:51:03.561857 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 00:51:03.563092 11629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29294bd0a7344b7a97e94ce57d16b6f3" candidate_uuid: "8e9d873796a2496ab0afae5438d5c425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efa11d703d04c938fc24412a7347bba" is_pre_election: true
I20260924 00:51:03.563380 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.556540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:56391 (local address 127.0.222.1:44623)
0924 00:51:03.556928 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.556934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.557222 (+ 288us) server_negotiation.cc:407] Connection header received
0924 00:51:03.557362 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.557367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.557447 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.557533 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:03.558086 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.559135 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.561305 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.562525 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.562642 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.562741 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.563006 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.563140 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260924 00:51:03.563402 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.556224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44623 (local address 127.0.222.3:56391)
0924 00:51:03.557108 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:03.557122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:03.557260 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:03.557591 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:03.557595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:03.557615 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:03.557928 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.557936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.559285 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.559288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.561121 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:03.561135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.562714 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.562720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:03.562861 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.562883 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 00:51:03.562959 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:51:03.563079 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":620,"threads_started":1}
I20260924 00:51:03.563634 11629 raft_consensus.cc:2454] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9d873796a2496ab0afae5438d5c425 in term 0.
I20260924 00:51:03.564392 11697 leader_election.cc:302] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1efa11d703d04c938fc24412a7347bba, 8e9d873796a2496ab0afae5438d5c425; no voters:
I20260924 00:51:03.564314 11494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29294bd0a7344b7a97e94ce57d16b6f3" candidate_uuid: "8e9d873796a2496ab0afae5438d5c425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321f617457ef4f64bca6808ac677c660" is_pre_election: true
I20260924 00:51:03.564795 11834 raft_consensus.cc:2781] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:51:03.564806 11494 raft_consensus.cc:2454] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e9d873796a2496ab0afae5438d5c425 in term 0.
I20260924 00:51:03.564952 11834 raft_consensus.cc:482] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:51:03.565059 11834 raft_consensus.cc:3037] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:51:03.567286 11834 raft_consensus.cc:504] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.567843 11834 leader_election.cc:288] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [CANDIDATE]: Term 1 election: Requested vote from peers 1efa11d703d04c938fc24412a7347bba (127.0.222.2:41175), 321f617457ef4f64bca6808ac677c660 (127.0.222.1:44623)
I20260924 00:51:03.568364 11629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29294bd0a7344b7a97e94ce57d16b6f3" candidate_uuid: "8e9d873796a2496ab0afae5438d5c425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1efa11d703d04c938fc24412a7347bba"
I20260924 00:51:03.568495 11494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29294bd0a7344b7a97e94ce57d16b6f3" candidate_uuid: "8e9d873796a2496ab0afae5438d5c425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321f617457ef4f64bca6808ac677c660"
I20260924 00:51:03.568552 11629 raft_consensus.cc:3037] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:51:03.568732 11494 raft_consensus.cc:3037] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:51:03.570890 11629 raft_consensus.cc:2454] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9d873796a2496ab0afae5438d5c425 in term 1.
I20260924 00:51:03.570910 11494 raft_consensus.cc:2454] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e9d873796a2496ab0afae5438d5c425 in term 1.
I20260924 00:51:03.571280 11697 leader_election.cc:302] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 321f617457ef4f64bca6808ac677c660, 8e9d873796a2496ab0afae5438d5c425; no voters:
I20260924 00:51:03.571583 11834 raft_consensus.cc:2781] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:51:03.572021 11834 raft_consensus.cc:686] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 1 LEADER]: Becoming Leader. State: Replica: 8e9d873796a2496ab0afae5438d5c425, State: Running, Role: LEADER
I20260924 00:51:03.572626 11834 consensus_queue.cc:237] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } }
I20260924 00:51:03.577190 11348 catalog_manager.cc:5495] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e9d873796a2496ab0afae5438d5c425 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "8e9d873796a2496ab0afae5438d5c425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e9d873796a2496ab0afae5438d5c425" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 38177 } health_report { overall_health: HEALTHY } } }
I20260924 00:51:03.615881 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.610311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.0.222.3:38177)
0924 00:51:03.610485 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.610489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.610506 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:51:03.610595 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.610599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.610686 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.610832 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:51:03.611376 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.612221 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.613722 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.614125 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.614225 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.614811 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:51:03.615325 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:51:03.615441 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.615546 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.615617 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:51:03.624145 11494 raft_consensus.cc:1261] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 1 FOLLOWER]: Refusing update from remote peer 8e9d873796a2496ab0afae5438d5c425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:51:03.624146 11629 raft_consensus.cc:1261] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 1 FOLLOWER]: Refusing update from remote peer 8e9d873796a2496ab0afae5438d5c425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:51:03.625116 11840 consensus_queue.cc:1035] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1efa11d703d04c938fc24412a7347bba" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:51:03.625537 11834 consensus_queue.cc:1035] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "321f617457ef4f64bca6808ac677c660" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:51:03.638175 11843 mvcc.cc:205] Tried to move back new op lower bound from 7332704516594855936 to 7332704516397748224. Current Snapshot: MvccSnapshot[applied={T|T < 7332704516594855936}]
I20260924 00:51:03.641350 11845 mvcc.cc:205] Tried to move back new op lower bound from 7332704516594855936 to 7332704516397748224. Current Snapshot: MvccSnapshot[applied={T|T < 7332704516594855936}]
I20260924 00:51:03.641633 11844 mvcc.cc:205] Tried to move back new op lower bound from 7332704516594855936 to 7332704516397748224. Current Snapshot: MvccSnapshot[applied={T|T < 7332704516594855936}]
I20260924 00:51:03.653585 11348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:51:03.653921 11348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:51:03.658079 11348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5df9cd2135847ddaa33b7f862005042: Sending DeleteTablet for 3 replicas of tablet 29294bd0a7344b7a97e94ce57d16b6f3
I20260924 00:51:03.659344 11609 tablet_service.cc:1514] Processing DeleteTablet for tablet 29294bd0a7344b7a97e94ce57d16b6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:51:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:34346
I20260924 00:51:03.659533 11474 tablet_service.cc:1514] Processing DeleteTablet for tablet 29294bd0a7344b7a97e94ce57d16b6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:51:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:35812
I20260924 00:51:03.659652 11744 tablet_service.cc:1514] Processing DeleteTablet for tablet 29294bd0a7344b7a97e94ce57d16b6f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:51:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:55244
I20260924 00:51:03.661134 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 11410
I20260924 00:51:03.670670 11853 tablet_replica.cc:330] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: stopping tablet replica
I20260924 00:51:03.671201 11853 raft_consensus.cc:2229] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:51:03.671587 11854 tablet_replica.cc:330] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: stopping tablet replica
I20260924 00:51:03.671803 11853 raft_consensus.cc:2258] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:51:03.672178 11854 raft_consensus.cc:2229] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:51:03.672659 11852 tablet_replica.cc:330] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: stopping tablet replica
I20260924 00:51:03.673007 11854 raft_consensus.cc:2258] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:51:03.673056 11852 raft_consensus.cc:2229] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:51:03.675202 11854 ts_tablet_manager.cc:1905] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:51:03.675195 11853 ts_tablet_manager.cc:1905] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:51:03.682153 11853 ts_tablet_manager.cc:1918] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:51:03.682370 11853 log.cc:1196] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/29294bd0a7344b7a97e94ce57d16b6f3
I20260924 00:51:03.682814 11854 ts_tablet_manager.cc:1918] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:51:03.682909 11853 ts_tablet_manager.cc:1939] T 29294bd0a7344b7a97e94ce57d16b6f3 P 321f617457ef4f64bca6808ac677c660: Deleting consensus metadata
I20260924 00:51:03.683447 11852 raft_consensus.cc:2258] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:51:03.683485 11854 log.cc:1196] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/29294bd0a7344b7a97e94ce57d16b6f3
I20260924 00:51:03.684196 11854 ts_tablet_manager.cc:1939] T 29294bd0a7344b7a97e94ce57d16b6f3 P 8e9d873796a2496ab0afae5438d5c425: Deleting consensus metadata
I20260924 00:51:03.684824 11334 catalog_manager.cc:4841] TS 321f617457ef4f64bca6808ac677c660 (127.0.222.1:44623): tablet 29294bd0a7344b7a97e94ce57d16b6f3 (table test-table [id=bff6cac420ab4a4d81b512d6b5fbdab6]) successfully deleted
I20260924 00:51:03.686172 11334 catalog_manager.cc:4841] TS 8e9d873796a2496ab0afae5438d5c425 (127.0.222.3:38177): tablet 29294bd0a7344b7a97e94ce57d16b6f3 (table test-table [id=bff6cac420ab4a4d81b512d6b5fbdab6]) successfully deleted
Sep 24 00:51:03 dist-test-slave-rtvz krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:51:03.686860 11852 ts_tablet_manager.cc:1905] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:51:03.692062 11852 ts_tablet_manager.cc:1918] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:51:03.692242 11852 log.cc:1196] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/29294bd0a7344b7a97e94ce57d16b6f3
I20260924 00:51:03.692801 11852 ts_tablet_manager.cc:1939] T 29294bd0a7344b7a97e94ce57d16b6f3 P 1efa11d703d04c938fc24412a7347bba: Deleting consensus metadata
I20260924 00:51:03.694468 11334 catalog_manager.cc:4841] TS 1efa11d703d04c938fc24412a7347bba (127.0.222.2:41175): tablet 29294bd0a7344b7a97e94ce57d16b6f3 (table test-table [id=bff6cac420ab4a4d81b512d6b5fbdab6]) successfully deleted
I20260924 00:51:03.695643 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.662824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.0.222.1:44623)
0924 00:51:03.663023 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.663030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.676172 (+ 13142us) server_negotiation.cc:407] Connection header received
0924 00:51:03.676633 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.676638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.676735 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.676809 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:03.678516 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.679659 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.681494 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.682105 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.682217 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.687852 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:03.687869 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:03.687888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:03.687936 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:03.691895 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.692870 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.692881 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.692885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.693081 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.693584 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.693590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.693593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.694134 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:03.694451 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.694946 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.695223 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:51:03.696424 11534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:03.786051 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 11410
I20260924 00:51:03.809319 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 11546
Sep 24 00:51:03 dist-test-slave-rtvz krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:51:03.826675 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.810052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.0.222.2:41175)
0924 00:51:03.810201 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.810205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.810283 (+ 78us) server_negotiation.cc:407] Connection header received
0924 00:51:03.810465 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.810469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.810542 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.810619 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:03.812240 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.812950 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.815057 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.815437 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.815527 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.820622 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:03.820640 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:03.820656 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:03.820695 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:03.823519 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.824509 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.824518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.824520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.824656 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.825077 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.825081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.825083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.825609 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:03.825853 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.826197 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.826397 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:51:03.827406 11670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:03.916265 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 11546
I20260924 00:51:03.930255 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 11680
Sep 24 00:51:03 dist-test-slave-rtvz krb5kdc[11303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:51:03.945783 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:03.931017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.0.222.3:38177)
0924 00:51:03.931192 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:03.931198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:03.931270 (+ 72us) server_negotiation.cc:407] Connection header received
0924 00:51:03.931446 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:03.931451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:03.931523 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:03.931607 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:03.932895 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.933866 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.935152 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:03.935591 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:03.935701 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:03.939896 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:03.939910 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:03.939926 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:03.939970 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:03.942881 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.943594 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.943602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.943604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.943787 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:03.944147 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:03.944151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:03.944153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:03.944516 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:03.944840 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:03.945108 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 00:51:03.945410 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:51:03.946442 11805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:04.068915 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 11680
I20260924 00:51:04.084699 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 11316
I20260924 00:51:04.085975 11379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:04.195214 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 11316
2026-09-24T00:51:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4137 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:51:04 dist-test-slave-rtvz krb5kdc[11862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:51:04 dist-test-slave-rtvz krb5kdc[11862](info): set up 2 sockets
Sep 24 00:51:04 dist-test-slave-rtvz krb5kdc[11862](info): commencing operation
krb5kdc: starting...
W20260924 00:51:05.225087 888 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:51:05 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:51:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/rangeradmin_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/rangeradmin_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/rangerlookup_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/rangerlookup_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
I20260924 00:51:05.546705 888 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfVll0B/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/postgres -l logfile start
2026-09-24 00:51:08.123 UTC [11893] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 00:51:08.123 UTC [11893] LOG: listening on IPv4 address "127.0.222.20", port 45729
2026-09-24 00:51:08.127 UTC [11893] LOG: listening on Unix socket "/tmp/.s.PGSQL.45729"
2026-09-24 00:51:08.149 UTC [11896] LOG: database system was shut down at 2026-09-24 00:51:06 UTC
2026-09-24 00:51:08.153 UTC [11893] LOG: database system is ready to accept connections
I20260924 00:51:09.045068 888 mini_postgres.cc:96] Postgres bound to 45729
2026-09-24 00:51:09.062 UTC [11904] FATAL: database "slave" does not exist
127.0.222.20:45729 - accepting connections
I20260924 00:51:09.063663 888 mini_ranger.cc:161] Starting Ranger...
I20260924 00:51:09.090559 888 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 00:51:09.572700 888 mini_ranger.cc:87] Created ranger Postgres database
I20260924 00:51:09.572875 888 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-admin
2026-09-24 00:51:09,884 [I] DB FLAVOR :POSTGRES
2026-09-24 00:51:09,884 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:51:09,884 [I] Checking connection..
2026-09-24 00:51:09,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 00:51:10,128 [I] Checking connection passed.
2026-09-24 00:51:10,128 [I] --------- Verifying version history table ---------
2026-09-24 00:51:10,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:51:10,399 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:51:10,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:51:10,661 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:51:10,661 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 00:51:10,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 00:51:10.903 UTC [11969] WARNING: there is no transaction in progress
2026-09-24 00:51:10,909 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 00:51:10,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:51:11,147 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 00:51:11,148 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:51:11,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 00:51:11,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 00:51:11,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rtvz.c.gcp-upstream.internal','N') ;"
2026-09-24 00:51:11,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:51:12,104 [I] Table x_portal_user does not exist in database ranger
2026-09-24 00:51:12,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:51:12,360 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 00:51:12,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:51:12,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:51:12,854 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 00:51:12,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 00:51:13.850 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:13.866 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:13.883 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:13.901 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:13.920 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:13.940 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.001 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.008 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.023 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.033 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.044 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.060 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.071 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.079 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.086 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14.094 UTC [12078] WARNING: there is no transaction in progress
2026-09-24 00:51:14,786 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 00:51:14,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 00:51:15,016 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 00:51:15,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:51:15,247 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 00:51:15,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:51:15,507 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 00:51:15,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:51:15,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:51:16,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rtvz.c.gcp-upstream.internal';"
2026-09-24 00:51:16,237 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 00:51:16,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 00:51:16,478 [I] DB_PATCHES have already been applied
I20260924 00:51:16.484593 888 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfVll0B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 00:51:16.484724 888 mini_ranger.cc:193] Using host: 127.0.222.20
I20260924 00:51:16.500032 888 mini_ranger.cc:227] Ranger admin URL: http://127.0.222.20:57965
Sep 24, 2026 12:51:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:51:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:51:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 12:51:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 12:51:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 12:51:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 12:51:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 00:51:17.566880 888 mini_ranger.cc:160] Time spent starting Ranger: real 8.503s user 0.002s sys 0.041s
Sep 24, 2026 12:51:38 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 12:51:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 12:51:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 12:51:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 12:51:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 00:51:40.283167 888 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/rangerkms_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/rangerkms_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.222.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.222.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 00:51:40 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211100, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 00:51:40.462436 888 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 00:51:40.484310 888 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 00:51:41.449937 888 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 00:51:41.450073 888 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms
I20260924 00:51:41.450259 888 mini_ranger_kms.cc:227] Using postgres at 127.0.222.20:45729
2026-09-24 00:51:41,974 [I] DB FLAVOR :POSTGRES
2026-09-24 00:51:41,974 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:51:41,974 [I] Checking connection
2026-09-24 00:51:41,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 00:51:42,219 [I] connection success
2026-09-24 00:51:42,220 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 00:51:42,220 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 00:51:42,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 00:51:42,461 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 00:51:42,461 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:51:42,461 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 00:51:42,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:45729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 00:51:42,728 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 00:51:43.227035 888 mini_ranger_kms.cc:310] Created kms service
I20260924 00:51:43.405670 888 mini_ranger_kms.cc:326] Created kudu user
I20260924 00:51:43.458652 888 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 00:51:43 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211100, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:51:43.919214 888 mini_ranger_kms.cc:384] Added ranger policy
I20260924 00:51:43.919551 888 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 00:51:43.919634 888 mini_ranger_kms.cc:240] Using host: 127.0.222.20
I20260924 00:51:43.940862 888 mini_ranger_kms.cc:276] Ranger KMS PID: 12263
I20260924 00:51:43.941087 888 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.222.20:57797
Sep 24, 2026 12:51:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:51:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:51:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 12:51:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 12:51:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 12:51:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 00:51:46.010576 888 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.548s user 0.134s sys 0.412s
I20260924 00:51:46.010798 888 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 00:51:46.010998 888 mini_ranger_kms.cc:401] 127.0.222.20:57797/kms/v1/keys
Sep 24 00:51:46 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 00:51:46 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Sep 24 00:51:47 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211106, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Sep 24 00:51:49 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:51:49.582756 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45953
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45953
--ranger_config_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:57797/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:51:50.185141 12314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:51:50.185572 12314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:51:50.185792 12314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:51:50.195767 12314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:51:50.195951 12314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:51:50.196044 12314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:51:50.196159 12314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:51:50.196259 12314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:51:50.209513 12314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.222.20:57797/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45953
--ranger_config_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:51:50.210942 12314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:51:50.212460 12314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:51:50.228107 12322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:50.228107 12319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:50.228199 12320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:50.229493 12314 server_base.cc:1034] running on GCE node
I20260924 00:51:50.230458 12314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:51:50.232275 12314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:51:50.233564 12314 hybrid_clock.cc:648] HybridClock initialized: now 1790211110233460 us; error 38 us; skew 500 ppm
I20260924 00:51:50.234031 12314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:51:50 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:51:50.247974 12314 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:51:50.250239 12314 webserver.cc:460] Webserver started at http://127.0.222.62:42539/ using document root <none> and password file <none>
I20260924 00:51:50.251092 12314 fs_manager.cc:359] Metadata directory not provided
I20260924 00:51:50.251292 12314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:51:50.251704 12314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:51:50 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:51:50.315013 12314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "d81f847968eb45f39af59cc2abd5928c"
format_stamp: "Formatted at 2026-09-24 00:51:50 on dist-test-slave-rtvz"
server_key: "0c7c23bd5dca63302230e68915fefcc5"
server_key_iv: "59af23e1fa5d2196b31bde609a37d34e"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:50.315970 12314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "d81f847968eb45f39af59cc2abd5928c"
format_stamp: "Formatted at 2026-09-24 00:51:50 on dist-test-slave-rtvz"
server_key: "0c7c23bd5dca63302230e68915fefcc5"
server_key_iv: "59af23e1fa5d2196b31bde609a37d34e"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:50.322492 12314 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 00:51:50.341459 12329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:50.343207 12314 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 00:51:50.343464 12314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "d81f847968eb45f39af59cc2abd5928c"
format_stamp: "Formatted at 2026-09-24 00:51:50 on dist-test-slave-rtvz"
server_key: "0c7c23bd5dca63302230e68915fefcc5"
server_key_iv: "59af23e1fa5d2196b31bde609a37d34e"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:50.343798 12314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:51:50.362236 12314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:51:50.367798 12314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:51:50.368243 12314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:51:50.408806 12314 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45953
I20260924 00:51:50.408869 12380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45953 every 8 connection(s)
I20260924 00:51:50.410544 12314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:51:50.415629 12381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:51:50.416509 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 12314
I20260924 00:51:50.416740 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:51:50.417722 888 external_mini_cluster.cc:1428] Setting key 2656099777e0491a081acca33fd4d6ef
I20260924 00:51:50.435232 12381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c: Bootstrap starting.
Sep 24 00:51:50 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:51:50.440197 12381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c: Neither blocks nor log segments found. Creating new log.
I20260924 00:51:50.441947 12381 log.cc:824] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:51:50.446274 12381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c: No bootstrap required, opened a new log
I20260924 00:51:50.447405 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:50.420690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.0.222.62:45953)
0924 00:51:50.422171 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:50.422184 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:50.422253 (+ 69us) server_negotiation.cc:407] Connection header received
0924 00:51:50.424976 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:50.425003 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:50.425560 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:50.425987 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:50.427746 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:50.429662 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:50.431575 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:50.432572 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:50.432737 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:50.437964 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:50.437976 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:50.437993 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:50.438055 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:50.442122 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:50.443120 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:50.443130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:50.443139 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:50.443375 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:50.443809 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:50.443814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:50.443817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:50.444494 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:50.444762 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:50.445431 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 00:51:50.445937 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1185,"thread_start_us":861,"threads_started":1}
I20260924 00:51:50.459932 12381 raft_consensus.cc:348] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81f847968eb45f39af59cc2abd5928c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45953 } }
I20260924 00:51:50.460747 12381 raft_consensus.cc:374] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:51:50.460925 12381 raft_consensus.cc:729] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81f847968eb45f39af59cc2abd5928c, State: Initialized, Role: FOLLOWER
I20260924 00:51:50.461874 12381 consensus_queue.cc:260] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81f847968eb45f39af59cc2abd5928c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45953 } }
I20260924 00:51:50.462322 12381 raft_consensus.cc:388] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:51:50.462549 12381 raft_consensus.cc:482] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:51:50.462831 12381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:51:50.467034 12381 raft_consensus.cc:504] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81f847968eb45f39af59cc2abd5928c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45953 } }
I20260924 00:51:50.467859 12381 leader_election.cc:302] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d81f847968eb45f39af59cc2abd5928c; no voters:
I20260924 00:51:50.468797 12381 leader_election.cc:288] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:51:50.468839 12386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:51:50.470161 12386 raft_consensus.cc:686] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [term 1 LEADER]: Becoming Leader. State: Replica: d81f847968eb45f39af59cc2abd5928c, State: Running, Role: LEADER
I20260924 00:51:50.470929 12386 consensus_queue.cc:237] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81f847968eb45f39af59cc2abd5928c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45953 } }
I20260924 00:51:50.472268 12381 sys_catalog.cc:564] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:51:50.475165 12387 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d81f847968eb45f39af59cc2abd5928c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81f847968eb45f39af59cc2abd5928c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45953 } } }
I20260924 00:51:50.476328 12387 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [sys.catalog]: This master's current role is: LEADER
I20260924 00:51:50.476347 12388 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d81f847968eb45f39af59cc2abd5928c. Latest consensus state: current_term: 1 leader_uuid: "d81f847968eb45f39af59cc2abd5928c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81f847968eb45f39af59cc2abd5928c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45953 } } }
I20260924 00:51:50.476673 12388 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c [sys.catalog]: This master's current role is: LEADER
I20260924 00:51:50.478821 12392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:51:50.486040 12392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:51:50.498085 12392 catalog_manager.cc:1347] Generated new cluster ID: 76c1364fb07348088b380f540a2b9f59
I20260924 00:51:50.498315 12392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:51:50.512070 12381 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 00:51:50.525847 12392 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:51:50.528134 12392 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:51:50.542026 12392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c: Generated new TSK 0
I20260924 00:51:50.542961 12392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:51:51.435105 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:57797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45953
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:51:51 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 00:51:52.055226 12427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:51:52.055622 12427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:51:52.055877 12427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:51:52.066773 12427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:51:52.066995 12427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:51:52.067225 12427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:51:52.081779 12427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.222.20:57797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:51:52.083343 12427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:51:52.084854 12427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:51:52.117952 12437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:52.118984 12438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:52.123840 12427 server_base.cc:1034] running on GCE node
W20260924 00:51:52.124275 12440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:52.125013 12427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:51:52.126349 12427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:51:52.127713 12427 hybrid_clock.cc:648] HybridClock initialized: now 1790211112127560 us; error 81 us; skew 500 ppm
I20260924 00:51:52.128389 12427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:51:52 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:51:52.134776 12427 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:51:52.141336 12427 webserver.cc:460] Webserver started at http://127.0.222.1:37535/ using document root <none> and password file <none>
I20260924 00:51:52.142642 12427 fs_manager.cc:359] Metadata directory not provided
I20260924 00:51:52.142891 12427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:51:52.143446 12427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:51:52 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:51:52.175905 12427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc"
format_stamp: "Formatted at 2026-09-24 00:51:52 on dist-test-slave-rtvz"
server_key: "cb4dc9cac848d33641664526aa725c08"
server_key_iv: "93f6c7de98749419a067af2e87429c79"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:52.190492 12427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc"
format_stamp: "Formatted at 2026-09-24 00:51:52 on dist-test-slave-rtvz"
server_key: "cb4dc9cac848d33641664526aa725c08"
server_key_iv: "93f6c7de98749419a067af2e87429c79"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:52.211292 12427 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.010s sys 0.000s
I20260924 00:51:52.254802 12447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:52.257259 12427 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.004s sys 0.004s
I20260924 00:51:52.257645 12427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc"
format_stamp: "Formatted at 2026-09-24 00:51:52 on dist-test-slave-rtvz"
server_key: "cb4dc9cac848d33641664526aa725c08"
server_key_iv: "93f6c7de98749419a067af2e87429c79"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:52.258116 12427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:51:52.306915 12427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:51:52.315146 12427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:51:52.315759 12427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:51:52.324220 12427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:51:52.333829 12427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:51:52.334095 12427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 00:51:52.334431 12427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:51:52.334620 12427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:51:52 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:51:52.617262 12427 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:34875
I20260924 00:51:52.617326 12560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:34875 every 8 connection(s)
I20260924 00:51:52.619406 12427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:51:52.622328 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 12427
I20260924 00:51:52.622565 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:51:52.623522 888 external_mini_cluster.cc:1428] Setting key e167e3e0e262f91c6b4c6f0c80587622
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Sep 24 00:51:52 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:51:52.673000 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:52.623397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:59093 (local address 127.0.222.62:45953)
0924 00:51:52.637944 (+ 14547us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:52.637951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:52.639334 (+ 1383us) server_negotiation.cc:407] Connection header received
0924 00:51:52.643262 (+ 3928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:52.643268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:52.645023 (+ 1755us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:52.645164 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:52.648515 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:52.650171 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:52.653673 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:52.654585 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:52.654705 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:52.663152 (+ 8447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:52.663163 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:52.663169 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:52.663229 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:52.668391 (+ 5162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:52.669677 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:52.669683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:52.669687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:52.669905 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:52.670743 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:52.670761 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:52.670764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:52.671199 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:52.671316 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:52.672204 (+ 888us) server_negotiation.cc:299] Negotiation successful
0924 00:51:52.672486 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14377,"thread_start_us":14150,"threads_started":1}
I20260924 00:51:52.674382 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:52.624154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45953 (local address 127.0.222.1:59093)
0924 00:51:52.638388 (+ 14234us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:52.638453 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:52.642898 (+ 4445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:52.645837 (+ 2939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:52.645853 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:52.646877 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:51:52.648239 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:52.648290 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:52.650515 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:52.650522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:52.653408 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:52.653471 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:52.655285 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:52.655295 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:52.655741 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:52.656737 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:51:52.656800 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:51:52.662669 (+ 5869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:51:52.668630 (+ 5961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:52.668642 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:52.668664 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:52.669514 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:52.670108 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:52.670113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:52.670117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:52.670599 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:52.671398 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:51:52.671404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:51:52.671781 (+ 377us) client_negotiation.cc:769] Sending connection context
0924 00:51:52.672055 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 00:51:52.672705 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13849,"thread_start_us":13407,"threads_started":1}
I20260924 00:51:52.676397 12561 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45953
I20260924 00:51:52.676783 12561 heartbeater.cc:464] Registering TS with master...
I20260924 00:51:52.678040 12561 heartbeater.cc:511] Master 127.0.222.62:45953 requested a full tablet report, sending...
I20260924 00:51:52.680979 12346 ts_manager.cc:194] Registered new tserver with Master: bc1ca009d21c4cfdb25cf8ae0faa6bdc (127.0.222.1:34875)
I20260924 00:51:52.683508 12346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:59093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:51:52.702111 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:57797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45953
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:51:52 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
W20260924 00:51:53.310883 12567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:51:53.311187 12567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:51:53.311334 12567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:51:53.320441 12567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:51:53.320592 12567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:51:53.320757 12567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:51:53.332512 12567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.222.20:57797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:51:53.333984 12567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:51:53.335376 12567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:51:53.350908 12582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:53.350894 12581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:53.353554 12584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:53.355444 12567 server_base.cc:1034] running on GCE node
I20260924 00:51:53.356272 12567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:51:53.357564 12567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:51:53.358829 12567 hybrid_clock.cc:648] HybridClock initialized: now 1790211113358756 us; error 53 us; skew 500 ppm
I20260924 00:51:53.359328 12567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:51:53 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:51:53.364286 12567 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:51:53.366189 12567 webserver.cc:460] Webserver started at http://127.0.222.2:34527/ using document root <none> and password file <none>
I20260924 00:51:53.366999 12567 fs_manager.cc:359] Metadata directory not provided
I20260924 00:51:53.367142 12567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:51:53.367470 12567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:51:53 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:51:53.389070 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "5c81b68c1a094b879521a24871676d41"
format_stamp: "Formatted at 2026-09-24 00:51:53 on dist-test-slave-rtvz"
server_key: "046adc5e6e8056dc495518f3af25d046"
server_key_iv: "5892ea5dac117be352adf7feaf136d6b"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:53.390355 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "5c81b68c1a094b879521a24871676d41"
format_stamp: "Formatted at 2026-09-24 00:51:53 on dist-test-slave-rtvz"
server_key: "046adc5e6e8056dc495518f3af25d046"
server_key_iv: "5892ea5dac117be352adf7feaf136d6b"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:53.398397 12567 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260924 00:51:53.420210 12591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:53.422333 12567 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 00:51:53.422574 12567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "5c81b68c1a094b879521a24871676d41"
format_stamp: "Formatted at 2026-09-24 00:51:53 on dist-test-slave-rtvz"
server_key: "046adc5e6e8056dc495518f3af25d046"
server_key_iv: "5892ea5dac117be352adf7feaf136d6b"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:53.422876 12567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:51:53.500955 12567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:51:53.505505 12567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:51:53.505805 12567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:51:53.506912 12567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:51:53.508621 12567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:51:53.508698 12567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:53.508829 12567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:51:53.508919 12567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:53.559047 12567 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:38181
I20260924 00:51:53.559064 12704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:38181 every 8 connection(s)
I20260924 00:51:53.560590 12567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:51:53.569619 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 12567
I20260924 00:51:53.569813 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:51:53.570341 888 external_mini_cluster.cc:1428] Setting key 2e40f67444aa7cf6637f32d9850ffa6c
Sep 24 00:51:53 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:51:53.606928 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:53.567545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:34857 (local address 127.0.222.62:45953)
0924 00:51:53.583467 (+ 15922us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:53.583474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:53.583498 (+ 24us) server_negotiation.cc:407] Connection header received
0924 00:51:53.585969 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:53.585975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:53.586107 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:53.586219 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:53.591929 (+ 5710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:53.593269 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:53.596025 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:53.596550 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:53.596758 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:53.600764 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:53.600770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:53.600773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:53.600817 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:53.603518 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:53.604339 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:53.604343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:53.604345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:53.604497 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:53.604950 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:53.604954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:53.604956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:53.605293 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:53.605423 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:53.606202 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 00:51:53.606419 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15787,"thread_start_us":15944,"threads_started":1}
I20260924 00:51:53.607882 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:53.568262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45953 (local address 127.0.222.2:34857)
0924 00:51:53.582719 (+ 14457us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:53.582787 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:53.585608 (+ 2821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:53.589766 (+ 4158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:53.589783 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:53.590585 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:51:53.591607 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:53.591656 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:53.593520 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:53.593527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:53.595793 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:53.595814 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:53.596917 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:53.596923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:53.597145 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:53.598042 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:51:53.598071 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:51:53.600410 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:51:53.603684 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:53.603699 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:53.603717 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:53.604188 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:53.604583 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:53.604586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:53.604588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:53.604823 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:53.605428 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:51:53.605435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:51:53.605801 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 00:51:53.606110 (+ 309us) client_negotiation.cc:240] Negotiation successful
0924 00:51:53.606578 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14015,"thread_start_us":15736,"threads_started":1}
I20260924 00:51:53.609702 12705 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45953
I20260924 00:51:53.610029 12705 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:51:53.610764 12705 heartbeater.cc:511] Master 127.0.222.62:45953 requested a full tablet report, sending...
I20260924 00:51:53.612483 12346 ts_manager.cc:194] Registered new tserver with Master: 5c81b68c1a094b879521a24871676d41 (127.0.222.2:38181)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:51:53.613441 12346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:34857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:51:53.644845 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:57797/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45953
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:51:53.688235 12561 heartbeater.cc:503] Master 127.0.222.62:45953 was elected leader, sending a full tablet report...
W20260924 00:51:54.090149 12711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:51:54.090453 12711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:51:54.090600 12711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:51:54.098289 12711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:51:54.098433 12711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:51:54.098599 12711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:51:54.108811 12711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.222.20:57797/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45953
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:51:54.110208 12711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:51:54.111549 12711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:51:54.124768 12719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:54.124847 12716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:51:54.124794 12717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:51:54.125838 12711 server_base.cc:1034] running on GCE node
I20260924 00:51:54.126442 12711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:51:54.127485 12711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:51:54.128859 12711 hybrid_clock.cc:648] HybridClock initialized: now 1790211114128793 us; error 48 us; skew 500 ppm
I20260924 00:51:54.129247 12711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:51:54 dist-test-slave-rtvz krb5kdc[11862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:51:54.135272 12711 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:51:54.136981 12711 webserver.cc:460] Webserver started at http://127.0.222.3:34325/ using document root <none> and password file <none>
I20260924 00:51:54.137769 12711 fs_manager.cc:359] Metadata directory not provided
I20260924 00:51:54.137898 12711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:51:54.138208 12711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:51:54 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:51:54.154533 12711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "7c7162a60e3c422da1eaaed24e51ddd0"
format_stamp: "Formatted at 2026-09-24 00:51:54 on dist-test-slave-rtvz"
server_key: "52dc80a86cddfa9329c6a55a5cc37b27"
server_key_iv: "123c51cd69c0eaf7789bc9b80a858280"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:54.155776 12711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "7c7162a60e3c422da1eaaed24e51ddd0"
format_stamp: "Formatted at 2026-09-24 00:51:54 on dist-test-slave-rtvz"
server_key: "52dc80a86cddfa9329c6a55a5cc37b27"
server_key_iv: "123c51cd69c0eaf7789bc9b80a858280"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:54.163262 12711 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 00:51:54.182090 12726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:54.184240 12711 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260924 00:51:54.184470 12711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "7c7162a60e3c422da1eaaed24e51ddd0"
format_stamp: "Formatted at 2026-09-24 00:51:54 on dist-test-slave-rtvz"
server_key: "52dc80a86cddfa9329c6a55a5cc37b27"
server_key_iv: "123c51cd69c0eaf7789bc9b80a858280"
server_key_version: "kuduclusterkey@0"
I20260924 00:51:54.184757 12711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:51:54.213039 12711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:51:54.219594 12711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:51:54.220182 12711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:51:54.221714 12711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:51:54.223980 12711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:51:54.224071 12711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:54.224216 12711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:51:54.224342 12711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:51:54.296859 12711 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:41527
I20260924 00:51:54.296917 12838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:41527 every 8 connection(s)
I20260924 00:51:54.298246 12711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:51:54.300619 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 12711
I20260924 00:51:54.300771 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:51:54.301397 888 external_mini_cluster.cc:1428] Setting key 78f6aa8246f7d0b903ec8f7076e9510d
Sep 24 00:51:54 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:51:54.324124 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.301488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:43529 (local address 127.0.222.62:45953)
0924 00:51:54.301996 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.302003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.302904 (+ 901us) server_negotiation.cc:407] Connection header received
0924 00:51:54.305029 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.305033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.305107 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.305242 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:54.307326 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.308091 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.310395 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.310855 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.310955 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.316006 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:54.316013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:54.316018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:54.316066 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:54.320344 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:54.321242 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:54.321247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:54.321250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:54.321423 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:54.322035 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:54.322040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:54.322044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:54.322361 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:54.322513 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.323284 (+ 771us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.323571 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":279,"threads_started":1}
I20260924 00:51:54.325474 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.301859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45953 (local address 127.0.222.3:43529)
0924 00:51:54.302759 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:54.302798 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:54.304746 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:54.305462 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:54.305471 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:54.306155 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:51:54.307140 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.307161 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.308294 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.308298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.310256 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.310268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.311259 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.311267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.311664 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.312456 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:51:54.312480 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:51:54.315714 (+ 3234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:51:54.320481 (+ 4767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:54.320491 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:54.320509 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:54.321089 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:54.321538 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:51:54.321543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:51:54.321546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:51:54.321906 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:51:54.322489 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:51:54.322497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:51:54.322863 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 00:51:54.323155 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 00:51:54.323861 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":398,"threads_started":1}
I20260924 00:51:54.327024 12839 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45953
I20260924 00:51:54.327550 12839 heartbeater.cc:464] Registering TS with master...
I20260924 00:51:54.328864 12839 heartbeater.cc:511] Master 127.0.222.62:45953 requested a full tablet report, sending...
I20260924 00:51:54.330787 12346 ts_manager.cc:194] Registered new tserver with Master: 7c7162a60e3c422da1eaaed24e51ddd0 (127.0.222.3:41527)
I20260924 00:51:54.331987 12346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:43529
I20260924 00:51:54.343206 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:51:54.368836 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.349468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.0.222.62:45953)
0924 00:51:54.349676 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.349680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.350697 (+ 1017us) server_negotiation.cc:407] Connection header received
0924 00:51:54.350954 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.350958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.351053 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.351168 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:54.353949 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.354931 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.357309 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.358076 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.358215 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.361575 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:54.361582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:54.361586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:54.361630 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:54.365084 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:54.366021 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:54.366025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:54.366027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:54.366232 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:54.366838 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:54.366845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:54.366848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:54.367335 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:54.367617 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.368012 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.368256 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 00:51:54.374176 12346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:51:54.383466 12346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:51:54.413946 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.399881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34875 (local address 127.0.0.1:34512)
0924 00:51:54.402138 (+ 2257us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:54.402180 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:54.403551 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:54.404902 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:54.404908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:54.404932 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:54.405841 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.405858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.408009 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.408015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.410291 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.410305 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.412844 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.412852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.413067 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.413135 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 00:51:54.413308 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 00:51:54.413529 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1950,"thread_start_us":1550,"threads_started":1}
I20260924 00:51:54.413978 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.400400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.0.222.1:34875)
0924 00:51:54.401464 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.401474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.402297 (+ 823us) server_negotiation.cc:407] Connection header received
0924 00:51:54.403927 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.403937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.404227 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.404423 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:54.406146 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.407841 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.410501 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.412057 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.412197 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.412587 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.413404 (+ 817us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.413600 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":701,"threads_started":1}
I20260924 00:51:54.415606 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.402001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:38181 (local address 127.0.0.1:46334)
0924 00:51:54.404003 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:54.404018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:54.404129 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:54.405242 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:54.405247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:54.405271 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:54.406694 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.406709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.408784 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.408789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.411877 (+ 3088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.411892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.414317 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.414324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.414556 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.414581 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 00:51:54.414670 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:51:54.414824 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1858,"mutex_wait_us":2,"thread_start_us":1515,"threads_started":1}
I20260924 00:51:54.416548 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.402518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.0.222.2:38181)
0924 00:51:54.403386 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.403392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.404264 (+ 872us) server_negotiation.cc:407] Connection header received
0924 00:51:54.404352 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.404358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.404599 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.404792 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:54.406941 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.408638 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.412046 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.413842 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.413973 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.414275 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.415838 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.416068 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":568,"threads_started":1}
I20260924 00:51:54.419450 12495 tablet_service.cc:1467] Processing CreateTablet for tablet 31c73d3388e44e488f1beb6ce901a041 (DEFAULT_TABLE table=test-table [id=82d95e877723423ba035489c91aefc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:51:54.421312 12495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c73d3388e44e488f1beb6ce901a041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:51:54.420747 12639 tablet_service.cc:1467] Processing CreateTablet for tablet 31c73d3388e44e488f1beb6ce901a041 (DEFAULT_TABLE table=test-table [id=82d95e877723423ba035489c91aefc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:51:54.422331 12639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c73d3388e44e488f1beb6ce901a041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:51:54.426126 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.400823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41527 (local address 127.0.0.1:43268)
0924 00:51:54.403746 (+ 2923us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:54.403798 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:54.404068 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:54.405318 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:54.405323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:54.405347 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:54.406811 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.406821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.412093 (+ 5272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.412098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.415327 (+ 3229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.415340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.425353 (+ 10013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.425359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.425514 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.425542 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 00:51:54.425633 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 00:51:54.425787 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2696,"thread_start_us":1711,"threads_started":1}
I20260924 00:51:54.426556 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.400401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.0.222.3:41527)
0924 00:51:54.402117 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.402126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.403920 (+ 1794us) server_negotiation.cc:407] Connection header received
0924 00:51:54.404264 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.404271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.404539 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.404745 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:54.407157 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.409348 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.415507 (+ 6159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.417634 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.417761 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.418091 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.425748 (+ 7657us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.425976 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1493,"thread_start_us":714,"threads_started":1}
I20260924 00:51:54.430995 12774 tablet_service.cc:1467] Processing CreateTablet for tablet 31c73d3388e44e488f1beb6ce901a041 (DEFAULT_TABLE table=test-table [id=82d95e877723423ba035489c91aefc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:51:54.433550 12774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31c73d3388e44e488f1beb6ce901a041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:51:54.437680 12858 tablet_bootstrap.cc:492] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Bootstrap starting.
I20260924 00:51:54.440755 12859 tablet_bootstrap.cc:492] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Bootstrap starting.
I20260924 00:51:54.445180 12858 tablet_bootstrap.cc:654] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Neither blocks nor log segments found. Creating new log.
I20260924 00:51:54.447034 12859 tablet_bootstrap.cc:654] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Neither blocks nor log segments found. Creating new log.
I20260924 00:51:54.447788 12858 log.cc:824] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Log is configured to *not* fsync() on all Append() calls
I20260924 00:51:54.449226 12859 log.cc:824] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Log is configured to *not* fsync() on all Append() calls
I20260924 00:51:54.451320 12863 tablet_bootstrap.cc:492] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Bootstrap starting.
I20260924 00:51:54.452634 12858 tablet_bootstrap.cc:492] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: No bootstrap required, opened a new log
I20260924 00:51:54.453086 12858 ts_tablet_manager.cc:1397] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20260924 00:51:54.455952 12859 tablet_bootstrap.cc:492] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: No bootstrap required, opened a new log
I20260924 00:51:54.456372 12859 ts_tablet_manager.cc:1397] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260924 00:51:54.459101 12863 tablet_bootstrap.cc:654] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Neither blocks nor log segments found. Creating new log.
I20260924 00:51:54.461603 12863 log.cc:824] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:51:54.467765 12858 raft_consensus.cc:348] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.468515 12858 raft_consensus.cc:374] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:51:54.468655 12858 raft_consensus.cc:729] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1ca009d21c4cfdb25cf8ae0faa6bdc, State: Initialized, Role: FOLLOWER
I20260924 00:51:54.469250 12863 tablet_bootstrap.cc:492] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: No bootstrap required, opened a new log
I20260924 00:51:54.469599 12863 ts_tablet_manager.cc:1397] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20260924 00:51:54.469499 12858 consensus_queue.cc:260] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.470052 12859 raft_consensus.cc:348] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.470647 12859 raft_consensus.cc:374] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:51:54.470888 12859 raft_consensus.cc:729] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c81b68c1a094b879521a24871676d41, State: Initialized, Role: FOLLOWER
I20260924 00:51:54.471786 12859 consensus_queue.cc:260] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.472548 12858 ts_tablet_manager.cc:1428] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
I20260924 00:51:54.474152 12705 heartbeater.cc:503] Master 127.0.222.62:45953 was elected leader, sending a full tablet report...
I20260924 00:51:54.475009 12859 ts_tablet_manager.cc:1428] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Time spent starting tablet: real 0.018s user 0.015s sys 0.005s
I20260924 00:51:54.484086 12863 raft_consensus.cc:348] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.484668 12863 raft_consensus.cc:374] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:51:54.484849 12863 raft_consensus.cc:729] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7162a60e3c422da1eaaed24e51ddd0, State: Initialized, Role: FOLLOWER
I20260924 00:51:54.485620 12863 consensus_queue.cc:260] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.486881 12839 heartbeater.cc:503] Master 127.0.222.62:45953 was elected leader, sending a full tablet report...
I20260924 00:51:54.487751 12863 ts_tablet_manager.cc:1428] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Time spent starting tablet: real 0.018s user 0.011s sys 0.006s
W20260924 00:51:54.550084 12840 tablet.cc:2369] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:51:54.563297 12706 tablet.cc:2369] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:51:54.598551 12867 raft_consensus.cc:482] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:51:54.598881 12867 raft_consensus.cc:504] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.600746 12867 leader_election.cc:288] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc1ca009d21c4cfdb25cf8ae0faa6bdc (127.0.222.1:34875), 5c81b68c1a094b879521a24871676d41 (127.0.222.2:38181)
I20260924 00:51:54.609390 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.601327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:35117 (local address 127.0.222.1:34875)
0924 00:51:54.601616 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.601623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.601906 (+ 283us) server_negotiation.cc:407] Connection header received
0924 00:51:54.602227 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.602233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.602300 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.602379 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:54.603090 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.604240 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.606970 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.608369 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.608528 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.608650 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.608929 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.609075 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 00:51:54.609383 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.601610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34875 (local address 127.0.222.3:35117)
0924 00:51:54.601791 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:54.601811 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:54.602035 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:54.602434 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:54.602439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:54.602473 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:54.602868 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.602883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.604377 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.604381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.606761 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.606776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.608562 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.608578 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.608755 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.608786 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 00:51:54.608867 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:51:54.608976 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 00:51:54.610809 12515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c73d3388e44e488f1beb6ce901a041" candidate_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" is_pre_election: true
I20260924 00:51:54.611673 12515 raft_consensus.cc:2454] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7162a60e3c422da1eaaed24e51ddd0 in term 0.
I20260924 00:51:54.611969 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.601638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:52615 (local address 127.0.222.2:38181)
0924 00:51:54.601890 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.601894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.603495 (+ 1601us) server_negotiation.cc:407] Connection header received
0924 00:51:54.603671 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.603675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.603737 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.603823 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:51:54.604730 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.606130 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.609285 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.610831 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.610966 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.611045 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.611527 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.611641 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:51:54.612044 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.602142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:38181 (local address 127.0.222.3:52615)
0924 00:51:54.603347 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0924 00:51:54.603382 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:51:54.603538 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:51:54.603846 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:51:54.603851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:51:54.603875 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:51:54.604543 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.604556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.606352 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.606356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.609104 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:51:54.609120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.611101 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.611109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:51:54.611323 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.611358 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:51:54.611459 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 00:51:54.611608 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"thread_start_us":830,"threads_started":1}
I20260924 00:51:54.612723 12727 leader_election.cc:302] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7162a60e3c422da1eaaed24e51ddd0, bc1ca009d21c4cfdb25cf8ae0faa6bdc; no voters:
I20260924 00:51:54.613287 12867 raft_consensus.cc:2781] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:51:54.613549 12867 raft_consensus.cc:482] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:51:54.613490 12659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c73d3388e44e488f1beb6ce901a041" candidate_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c81b68c1a094b879521a24871676d41" is_pre_election: true
I20260924 00:51:54.613734 12867 raft_consensus.cc:3037] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:51:54.614113 12659 raft_consensus.cc:2454] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7162a60e3c422da1eaaed24e51ddd0 in term 0.
I20260924 00:51:54.617316 12867 raft_consensus.cc:504] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.618279 12867 leader_election.cc:288] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [CANDIDATE]: Term 1 election: Requested vote from peers bc1ca009d21c4cfdb25cf8ae0faa6bdc (127.0.222.1:34875), 5c81b68c1a094b879521a24871676d41 (127.0.222.2:38181)
I20260924 00:51:54.618793 12515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c73d3388e44e488f1beb6ce901a041" candidate_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc"
I20260924 00:51:54.618947 12659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c73d3388e44e488f1beb6ce901a041" candidate_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c81b68c1a094b879521a24871676d41"
I20260924 00:51:54.619073 12515 raft_consensus.cc:3037] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:51:54.619266 12659 raft_consensus.cc:3037] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:51:54.623704 12515 raft_consensus.cc:2454] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7162a60e3c422da1eaaed24e51ddd0 in term 1.
I20260924 00:51:54.624126 12659 raft_consensus.cc:2454] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7162a60e3c422da1eaaed24e51ddd0 in term 1.
I20260924 00:51:54.624421 12727 leader_election.cc:302] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c7162a60e3c422da1eaaed24e51ddd0, bc1ca009d21c4cfdb25cf8ae0faa6bdc; no voters:
I20260924 00:51:54.624836 12867 raft_consensus.cc:2781] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 00:51:54.625645 12562 tablet.cc:2369] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:51:54.625877 12867 raft_consensus.cc:686] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7162a60e3c422da1eaaed24e51ddd0, State: Running, Role: LEADER
I20260924 00:51:54.626658 12867 consensus_queue.cc:237] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } }
I20260924 00:51:54.632246 12346 catalog_manager.cc:5495] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c7162a60e3c422da1eaaed24e51ddd0 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c7162a60e3c422da1eaaed24e51ddd0" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 } health_report { overall_health: UNKNOWN } } }
I20260924 00:51:54.655838 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.649830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.0.222.3:41527)
0924 00:51:54.650100 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.650105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.650126 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:51:54.650228 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.650233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.650322 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.650473 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:51:54.651140 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.651961 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.653696 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.654091 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.654185 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.654796 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:51:54.655311 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:51:54.655417 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.655519 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.655588 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 00:51:54.666785 12659 raft_consensus.cc:1261] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 1 FOLLOWER]: Refusing update from remote peer 7c7162a60e3c422da1eaaed24e51ddd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:51:54.667068 12515 raft_consensus.cc:1261] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 1 FOLLOWER]: Refusing update from remote peer 7c7162a60e3c422da1eaaed24e51ddd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:51:54.667948 12869 consensus_queue.cc:1035] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c81b68c1a094b879521a24871676d41" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 38181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:51:54.668510 12867 consensus_queue.cc:1035] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc1ca009d21c4cfdb25cf8ae0faa6bdc" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:51:54.682417 12872 mvcc.cc:205] Tried to move back new op lower bound from 7332704725663264768 to 7332704725515935744. Current Snapshot: MvccSnapshot[applied={T|T < 7332704725663264768}]
I20260924 00:51:54.701566 12346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:51:54.702013 12346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:51:54.706740 12346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d81f847968eb45f39af59cc2abd5928c: Sending DeleteTablet for 3 replicas of tablet 31c73d3388e44e488f1beb6ce901a041
I20260924 00:51:54.708387 12774 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c73d3388e44e488f1beb6ce901a041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:51:54 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:43268
I20260924 00:51:54.708408 12495 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c73d3388e44e488f1beb6ce901a041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:51:54 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260924 00:51:54.708609 12639 tablet_service.cc:1514] Processing DeleteTablet for tablet 31c73d3388e44e488f1beb6ce901a041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:51:54 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:46334
I20260924 00:51:54.715791 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 12427
I20260924 00:51:54.724040 12882 tablet_replica.cc:330] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: stopping tablet replica
I20260924 00:51:54.724754 12882 raft_consensus.cc:2229] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:51:54.725394 12883 tablet_replica.cc:330] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: stopping tablet replica
I20260924 00:51:54.725611 12882 raft_consensus.cc:2258] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:51:54.726056 12883 raft_consensus.cc:2229] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:51:54.728123 12882 ts_tablet_manager.cc:1905] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:51:54.728670 12884 tablet_replica.cc:330] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: stopping tablet replica
I20260924 00:51:54.729223 12884 raft_consensus.cc:2229] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:51:54.730113 12884 raft_consensus.cc:2258] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:51:54.732494 12884 ts_tablet_manager.cc:1905] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:51:54.733624 12883 raft_consensus.cc:2258] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:51:54.736913 12883 ts_tablet_manager.cc:1905] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:51:54.738610 12882 ts_tablet_manager.cc:1918] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:51:54.738862 12882 log.cc:1196] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/31c73d3388e44e488f1beb6ce901a041
I20260924 00:51:54.739470 12882 ts_tablet_manager.cc:1939] T 31c73d3388e44e488f1beb6ce901a041 P bc1ca009d21c4cfdb25cf8ae0faa6bdc: Deleting consensus metadata
Sep 24 00:51:54 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:51:54.741835 12330 catalog_manager.cc:4841] TS bc1ca009d21c4cfdb25cf8ae0faa6bdc (127.0.222.1:34875): tablet 31c73d3388e44e488f1beb6ce901a041 (table test-table [id=82d95e877723423ba035489c91aefc53]) successfully deleted
I20260924 00:51:54.742712 12884 ts_tablet_manager.cc:1918] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:51:54.742974 12884 log.cc:1196] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/31c73d3388e44e488f1beb6ce901a041
I20260924 00:51:54.743755 12884 ts_tablet_manager.cc:1939] T 31c73d3388e44e488f1beb6ce901a041 P 5c81b68c1a094b879521a24871676d41: Deleting consensus metadata
I20260924 00:51:54.746325 12331 catalog_manager.cc:4841] TS 5c81b68c1a094b879521a24871676d41 (127.0.222.2:38181): tablet 31c73d3388e44e488f1beb6ce901a041 (table test-table [id=82d95e877723423ba035489c91aefc53]) successfully deleted
I20260924 00:51:54.752846 12883 ts_tablet_manager.cc:1918] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:51:54.753334 12883 log.cc:1196] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/31c73d3388e44e488f1beb6ce901a041
I20260924 00:51:54.754503 12883 ts_tablet_manager.cc:1939] T 31c73d3388e44e488f1beb6ce901a041 P 7c7162a60e3c422da1eaaed24e51ddd0: Deleting consensus metadata
I20260924 00:51:54.754611 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.720748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.0.222.1:34875)
0924 00:51:54.720974 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.720980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.727433 (+ 6453us) server_negotiation.cc:407] Connection header received
0924 00:51:54.727639 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.727644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.727732 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.727849 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:54.729726 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.730902 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.733112 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.734144 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.734277 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.744236 (+ 9959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:54.744254 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:54.744274 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:54.744372 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:54.750349 (+ 5977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:54.751287 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:54.751295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:54.751297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:54.751496 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:54.752053 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:54.752057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:54.752059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:54.752831 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:54.753369 (+ 538us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.753691 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.753971 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 00:51:54.755530 12556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:54.760953 12333 catalog_manager.cc:4841] TS 7c7162a60e3c422da1eaaed24e51ddd0 (127.0.222.3:41527): tablet 31c73d3388e44e488f1beb6ce901a041 (table test-table [id=82d95e877723423ba035489c91aefc53]) successfully deleted
I20260924 00:51:54.856001 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 12427
I20260924 00:51:54.876773 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 12567
Sep 24 00:51:54 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:51:54.895025 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:54.877557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.0.222.2:38181)
0924 00:51:54.877817 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:54.877823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:54.877854 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:51:54.878057 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:54.878063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:54.878154 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:54.878242 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:54.879762 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.880859 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.882597 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:54.883247 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:54.883393 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:54.887833 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:54.887856 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:54.887877 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:54.887934 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:54.892036 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:54.892733 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:54.892743 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:54.892747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:54.892911 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:54.893420 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:54.893425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:54.893429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:54.893914 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:54.894206 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:54.894423 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 00:51:54.894658 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:51:54.895780 12700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:54.986405 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 12567
I20260924 00:51:55.000769 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 12711
Sep 24 00:51:55 dist-test-slave-rtvz krb5kdc[11862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:51:55.017923 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:51:55.001557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.0.222.3:41527)
0924 00:51:55.001738 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 00:51:55.001744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:51:55.001877 (+ 133us) server_negotiation.cc:407] Connection header received
0924 00:51:55.002020 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:51:55.002026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:51:55.002094 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:51:55.002181 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:51:55.003541 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:55.004371 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:55.005769 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:51:55.006224 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:51:55.006346 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:51:55.010978 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:51:55.010994 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:51:55.011015 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:51:55.011072 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:51:55.014982 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:55.015689 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:55.015699 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:55.015703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:55.015898 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:51:55.016271 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:51:55.016276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:51:55.016279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:51:55.016735 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:51:55.017042 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 00:51:55.017322 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 00:51:55.017538 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 00:51:55.018676 12834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:55.136147 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 12711
I20260924 00:51:55.152477 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 12314
I20260924 00:51:55.153751 12376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:51:55.267925 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 12314
I20260924 00:51:55.286151 888 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 00:51:55.622010 888 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 00:51:55.622319 888 mini_ranger.cc:66] Stopping Ranger...
I20260924 00:51:55.958630 888 mini_ranger.cc:68] Stopped Ranger
2026-09-24 00:51:55.959 UTC [11893] LOG: received smart shutdown request
2026-09-24 00:51:55.962 UTC [11893] LOG: background worker "logical replication launcher" (PID 11902) exited with exit code 1
2026-09-24 00:51:55.963 UTC [11897] LOG: shutting down
2026-09-24 00:51:56.173 UTC [11893] LOG: database system is shut down
2026-09-24T00:51:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52087 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:51:56 dist-test-slave-rtvz krb5kdc[12898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:51:56 dist-test-slave-rtvz krb5kdc[12898](info): set up 2 sockets
Sep 24 00:51:56 dist-test-slave-rtvz krb5kdc[12898](info): commencing operation
krb5kdc: starting...
W20260924 00:51:57.336938 888 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:51:57 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:51:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/rangeradmin_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/rangeradmin_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/rangerlookup_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/rangerlookup_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
I20260924 00:51:57.655967 888 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfVll0B/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/postgres -l logfile start
2026-09-24 00:52:00.521 UTC [12928] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 00:52:00.522 UTC [12928] LOG: listening on IPv4 address "127.0.222.20", port 58217
2026-09-24 00:52:00.525 UTC [12928] LOG: listening on Unix socket "/tmp/.s.PGSQL.58217"
2026-09-24 00:52:00.548 UTC [12931] LOG: database system was shut down at 2026-09-24 00:51:58 UTC
2026-09-24 00:52:00.551 UTC [12928] LOG: database system is ready to accept connections
I20260924 00:52:01.468437 888 mini_postgres.cc:96] Postgres bound to 58217
2026-09-24 00:52:01.484 UTC [12940] FATAL: database "slave" does not exist
127.0.222.20:58217 - accepting connections
I20260924 00:52:01.485337 888 mini_ranger.cc:161] Starting Ranger...
I20260924 00:52:01.511339 888 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 00:52:02.044737 888 mini_ranger.cc:87] Created ranger Postgres database
I20260924 00:52:02.044853 888 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-admin
2026-09-24 00:52:02,340 [I] DB FLAVOR :POSTGRES
2026-09-24 00:52:02,340 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:52:02,340 [I] Checking connection..
2026-09-24 00:52:02,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 00:52:02,571 [I] Checking connection passed.
2026-09-24 00:52:02,572 [I] --------- Verifying version history table ---------
2026-09-24 00:52:02,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:52:02,837 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:52:02,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:52:03,079 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:52:03,079 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 00:52:03,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 00:52:03.324 UTC [13004] WARNING: there is no transaction in progress
2026-09-24 00:52:03,331 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 00:52:03,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:52:03,559 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 00:52:03,560 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:52:03,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 00:52:03,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 00:52:04,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rtvz.c.gcp-upstream.internal','N') ;"
2026-09-24 00:52:04,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:52:04,546 [I] Table x_portal_user does not exist in database ranger
2026-09-24 00:52:04,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:52:04,811 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 00:52:04,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:52:05,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:52:05,321 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 00:52:05,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 00:52:06.294 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.312 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.327 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.344 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.361 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.379 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.439 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.447 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.463 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.473 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.484 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.498 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.508 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.514 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.520 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:06.527 UTC [13113] WARNING: there is no transaction in progress
2026-09-24 00:52:07,248 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 00:52:07,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 00:52:07,506 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 00:52:07,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:52:07,743 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 00:52:07,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:52:07,991 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 00:52:07,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:52:08,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:52:08,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rtvz.c.gcp-upstream.internal';"
2026-09-24 00:52:08,694 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 00:52:08,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 00:52:08,940 [I] DB_PATCHES have already been applied
I20260924 00:52:08.945820 888 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfVll0B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 00:52:08.945925 888 mini_ranger.cc:193] Using host: 127.0.222.20
I20260924 00:52:08.959856 888 mini_ranger.cc:227] Ranger admin URL: http://127.0.222.20:37195
Sep 24, 2026 12:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 12:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 12:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 12:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 12:52:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 00:52:10.004554 888 mini_ranger.cc:160] Time spent starting Ranger: real 8.519s user 0.006s sys 0.035s
Sep 24, 2026 12:52:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 12:52:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 12:52:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 12:52:30 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 12:52:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 00:52:31.645732 888 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.222.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.222.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/rangerkms_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.222.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/rangerkms_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.222.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.222.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.222.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/HTTP_127.0.222.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 00:52:31 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211151, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 00:52:31.826963 888 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 00:52:31.850399 888 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 00:52:32.785461 888 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 00:52:32.785578 888 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms
I20260924 00:52:32.785621 888 mini_ranger_kms.cc:227] Using postgres at 127.0.222.20:58217
2026-09-24 00:52:33,138 [I] DB FLAVOR :POSTGRES
2026-09-24 00:52:33,139 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:52:33,139 [I] Checking connection
2026-09-24 00:52:33,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 00:52:33,409 [I] connection success
2026-09-24 00:52:33,409 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 00:52:33,409 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 00:52:33,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 00:52:33,672 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 00:52:33,673 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:52:33,673 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 00:52:33,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.222.20:58217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 00:52:33,941 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 00:52:34.460464 888 mini_ranger_kms.cc:310] Created kms service
I20260924 00:52:34.685869 888 mini_ranger_kms.cc:326] Created kudu user
I20260924 00:52:34.733392 888 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 00:52:34 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211151, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:35.152194 888 mini_ranger_kms.cc:384] Added ranger policy
I20260924 00:52:35.152515 888 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfVll0B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfVll0B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 00:52:35.152565 888 mini_ranger_kms.cc:240] Using host: 127.0.222.20
I20260924 00:52:35.170102 888 mini_ranger_kms.cc:276] Ranger KMS PID: 13295
I20260924 00:52:35.170238 888 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.222.20:60711
Sep 24, 2026 12:52:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:52:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:52:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 12:52:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 12:52:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 12:52:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 00:52:37.248059 888 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.421s user 0.062s sys 0.316s
I20260924 00:52:37.248258 888 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 00:52:37.248412 888 mini_ranger_kms.cc:401] 127.0.222.20:60711/kms/v1/keys
Sep 24 00:52:37 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 00:52:37 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Sep 24 00:52:38 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.222.20@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Sep 24 00:52:41 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:52:41.550033 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:35035
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:35035
--ranger_config_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:60711/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:52:41.958863 13346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:41.959209 13346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:41.959358 13346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:41.967355 13346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:52:41.967489 13346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:41.967569 13346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:41.967644 13346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:52:41.967710 13346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:52:41.978709 13346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.222.20:60711/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:35035
--ranger_config_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:41.980223 13346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:41.981667 13346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:41.993999 13353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:41.994076 13351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:41.994050 13355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:41.994483 13346 server_base.cc:1034] running on GCE node
I20260924 00:52:41.995369 13346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:41.996873 13346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:41.998073 13346 hybrid_clock.cc:648] HybridClock initialized: now 1790211161998033 us; error 34 us; skew 500 ppm
I20260924 00:52:41.998476 13346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:41 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:42.001652 13346 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:52:42.003197 13346 webserver.cc:460] Webserver started at http://127.0.222.62:41091/ using document root <none> and password file <none>
I20260924 00:52:42.003854 13346 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:42.003949 13346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:42.004225 13346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:52:42 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:42.057003 13346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "5b6ef0863cf94b21950e861711d69ff2"
format_stamp: "Formatted at 2026-09-24 00:52:42 on dist-test-slave-rtvz"
server_key: "5b01f0254222e0e3e4de898243624375"
server_key_iv: "c2a259c45f4b1ba068fd0211708473ba"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:42.058197 13346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "5b6ef0863cf94b21950e861711d69ff2"
format_stamp: "Formatted at 2026-09-24 00:52:42 on dist-test-slave-rtvz"
server_key: "5b01f0254222e0e3e4de898243624375"
server_key_iv: "c2a259c45f4b1ba068fd0211708473ba"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:42.064759 13346 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260924 00:52:42.087888 13362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:42.089774 13346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 00:52:42.089996 13346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "5b6ef0863cf94b21950e861711d69ff2"
format_stamp: "Formatted at 2026-09-24 00:52:42 on dist-test-slave-rtvz"
server_key: "5b01f0254222e0e3e4de898243624375"
server_key_iv: "c2a259c45f4b1ba068fd0211708473ba"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:42.090312 13346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:42.108829 13346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:42.114135 13346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:42.114494 13346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:42.134182 13346 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:35035
I20260924 00:52:42.134195 13413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:35035 every 8 connection(s)
I20260924 00:52:42.135983 13346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:52:42.139526 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 13346
I20260924 00:52:42.139750 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:52:42.140640 888 external_mini_cluster.cc:1428] Setting key 712bda0f6808cac9cef4a3a86948695f
I20260924 00:52:42.140792 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:52:42.157841 13414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Bootstrap starting.
I20260924 00:52:42.161278 13414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:42.162595 13414 log.cc:824] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:42.166167 13414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: No bootstrap required, opened a new log
Sep 24 00:52:42 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:42.174907 13414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } }
I20260924 00:52:42.175344 13414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:42.175495 13414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6ef0863cf94b21950e861711d69ff2, State: Initialized, Role: FOLLOWER
I20260924 00:52:42.176194 13414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } }
I20260924 00:52:42.176429 13414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:52:42.176586 13414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:52:42.176759 13414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:52:42.179503 13414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } }
I20260924 00:52:42.180111 13414 leader_election.cc:302] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b6ef0863cf94b21950e861711d69ff2; no voters:
I20260924 00:52:42.181299 13414 leader_election.cc:288] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:52:42.183466 13417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:42.142895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.0.222.62:35035)
0924 00:52:42.157836 (+ 14941us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:42.157855 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:42.157914 (+ 59us) server_negotiation.cc:407] Connection header received
0924 00:52:42.160615 (+ 2701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:42.160647 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:42.161283 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:42.161840 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:42.163525 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:42.165412 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:42.167360 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:42.168413 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:42.168597 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:42.173550 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:42.173567 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:42.173596 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:42.173654 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:42.177999 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:42.179052 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:42.179062 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:42.179077 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:42.179316 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:42.179864 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:42.179870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:42.179874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:42.180672 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:42.180990 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:42.181572 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 00:52:42.182034 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":14644,"thread_start_us":14394,"threads_started":1}
I20260924 00:52:42.184017 13414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:52:42.184285 13419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:52:42.191318 13419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 1 LEADER]: Becoming Leader. State: Replica: 5b6ef0863cf94b21950e861711d69ff2, State: Running, Role: LEADER
I20260924 00:52:42.191994 13419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } }
I20260924 00:52:42.195001 13420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b6ef0863cf94b21950e861711d69ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } } }
I20260924 00:52:42.195426 13420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:42.196542 13421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b6ef0863cf94b21950e861711d69ff2. Latest consensus state: current_term: 1 leader_uuid: "5b6ef0863cf94b21950e861711d69ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } } }
I20260924 00:52:42.197773 13421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:42.199225 13426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:52:42.204124 13426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:52:42.204873 13414 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 00:52:42.222375 13426 catalog_manager.cc:1347] Generated new cluster ID: b8b4e271660343b4b06bca26ff6f8fcb
I20260924 00:52:42.222532 13426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:52:42.240536 13426 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:52:42.243016 13426 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:52:42.259764 13426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Generated new TSK 0
I20260924 00:52:42.260984 13426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:52:43.014514 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:60711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:35035
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:52:43 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 00:52:43.456799 13459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:43.457340 13459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:43.457561 13459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:43.469744 13459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:43.469904 13459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:43.470147 13459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:52:43.486639 13459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.222.20:60711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:43.488474 13459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:43.490221 13459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:43.510144 13469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:43.522122 13470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:43.525744 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:43.526432 13459 server_base.cc:1034] running on GCE node
I20260924 00:52:43.527263 13459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:43.529338 13459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:43.532802 13459 hybrid_clock.cc:648] HybridClock initialized: now 1790211163532699 us; error 73 us; skew 500 ppm
I20260924 00:52:43.533378 13459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:43 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:43.538332 13459 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:52:43.540458 13459 webserver.cc:460] Webserver started at http://127.0.222.1:32883/ using document root <none> and password file <none>
I20260924 00:52:43.541352 13459 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:43.541505 13459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:43.541862 13459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:52:43 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:43.561794 13459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "a91ac1681a034147b47af97f6df8a0e6"
format_stamp: "Formatted at 2026-09-24 00:52:43 on dist-test-slave-rtvz"
server_key: "0f1db334d9815f4bda8f925ae0eeb4c6"
server_key_iv: "b68c15be21fb22a1ad5e00988ce2c401"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:43.563074 13459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "a91ac1681a034147b47af97f6df8a0e6"
format_stamp: "Formatted at 2026-09-24 00:52:43 on dist-test-slave-rtvz"
server_key: "0f1db334d9815f4bda8f925ae0eeb4c6"
server_key_iv: "b68c15be21fb22a1ad5e00988ce2c401"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:43.571393 13459 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260924 00:52:43.605914 13480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:43.617585 13459 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.003s sys 0.012s
I20260924 00:52:43.617933 13459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "a91ac1681a034147b47af97f6df8a0e6"
format_stamp: "Formatted at 2026-09-24 00:52:43 on dist-test-slave-rtvz"
server_key: "0f1db334d9815f4bda8f925ae0eeb4c6"
server_key_iv: "b68c15be21fb22a1ad5e00988ce2c401"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:43.618367 13459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:43.643419 13459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:43.649042 13459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:43.649436 13459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:43.650708 13459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:52:43.652561 13459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:52:43.652652 13459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:43.652805 13459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:52:43.652872 13459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:43.722970 13459 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44719
I20260924 00:52:43.723505 13593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44719 every 8 connection(s)
I20260924 00:52:43.724905 13459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:52:43.727391 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 13459
I20260924 00:52:43.727620 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:52:43.728652 888 external_mini_cluster.cc:1428] Setting key 2537991ef3ab7561f0a5b870cac49eec
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Sep 24 00:52:43 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
Sep 24 00:52:43 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:43.809942 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:43.749221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55257 (local address 127.0.222.62:35035)
0924 00:52:43.775597 (+ 26376us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:43.775604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:43.775629 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:52:43.777374 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:43.777380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:43.777491 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:43.777589 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:43.779931 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:43.781131 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:43.783682 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:43.785523 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:43.785675 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:43.799364 (+ 13689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:43.799372 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:43.799376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:43.799420 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:43.802905 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:43.803939 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:43.803944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:43.803948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:43.804114 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:43.804707 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:43.804712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:43.804715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:43.806637 (+ 1922us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:43.806807 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:43.809177 (+ 2370us) server_negotiation.cc:299] Negotiation successful
0924 00:52:43.809482 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26240,"thread_start_us":27745,"threads_started":1}
I20260924 00:52:43.812206 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:43.750067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:35035 (local address 127.0.222.1:55257)
0924 00:52:43.774486 (+ 24419us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:43.774546 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:43.776872 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:43.778073 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:43.778086 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:43.778795 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:43.779697 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:43.779724 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:43.781368 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:43.781375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:43.783461 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:43.783479 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:43.790168 (+ 6689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:43.790177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:43.790534 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:43.795631 (+ 5097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:43.795723 (+ 92us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:43.798966 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:43.803105 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:43.803115 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:43.803136 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:43.803787 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:43.804254 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:43.804259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:43.804263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:43.804569 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:43.808047 (+ 3478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:43.808059 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:43.808609 (+ 550us) client_negotiation.cc:769] Sending connection context
0924 00:52:43.810299 (+ 1690us) client_negotiation.cc:240] Negotiation successful
0924 00:52:43.810948 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24052,"thread_start_us":23815,"threads_started":1}
I20260924 00:52:43.817452 13594 heartbeater.cc:347] Connected to a master server at 127.0.222.62:35035
I20260924 00:52:43.817932 13594 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:43.819315 13594 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
I20260924 00:52:43.822409 13379 ts_manager.cc:194] Registered new tserver with Master: a91ac1681a034147b47af97f6df8a0e6 (127.0.222.1:44719)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:52:43.824635 13379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:55257
I20260924 00:52:43.826036 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:60711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:35035
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:52:44 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
W20260924 00:52:44.353871 13600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:44.354411 13600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:44.354615 13600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:44.366236 13600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:44.366466 13600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:44.366703 13600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:52:44.382905 13600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.222.20:60711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:44.384960 13600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:44.386785 13600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:44.424825 13607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:44.425849 13608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:44.439543 13611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:44.440068 13600 server_base.cc:1034] running on GCE node
I20260924 00:52:44.441447 13600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:44.453575 13600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:44.454998 13600 hybrid_clock.cc:648] HybridClock initialized: now 1790211164454882 us; error 124 us; skew 500 ppm
I20260924 00:52:44.455528 13600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:44 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:44.460177 13600 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:52:44.481298 13600 webserver.cc:460] Webserver started at http://127.0.222.2:40921/ using document root <none> and password file <none>
I20260924 00:52:44.482646 13600 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:44.482910 13600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:44.483381 13600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:52:44 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:44.527060 13600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "2defc331ddf14257bbeef3cab789a55b"
format_stamp: "Formatted at 2026-09-24 00:52:44 on dist-test-slave-rtvz"
server_key: "9aca84766d7142678b190871482dbc52"
server_key_iv: "6b71ddff6949da7b812b54e123175746"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:44.530113 13600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "2defc331ddf14257bbeef3cab789a55b"
format_stamp: "Formatted at 2026-09-24 00:52:44 on dist-test-slave-rtvz"
server_key: "9aca84766d7142678b190871482dbc52"
server_key_iv: "6b71ddff6949da7b812b54e123175746"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:44.551983 13600 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260924 00:52:44.575063 13624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:44.576969 13600 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 00:52:44.577284 13600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "2defc331ddf14257bbeef3cab789a55b"
format_stamp: "Formatted at 2026-09-24 00:52:44 on dist-test-slave-rtvz"
server_key: "9aca84766d7142678b190871482dbc52"
server_key_iv: "6b71ddff6949da7b812b54e123175746"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:44.577654 13600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:44.614704 13600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:44.620605 13600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:44.621043 13600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:44.622845 13600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:52:44.625047 13600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:52:44.625249 13600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:44.625450 13600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:52:44.625566 13600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:44.666142 13600 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:44445
I20260924 00:52:44.666177 13737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:44445 every 8 connection(s)
I20260924 00:52:44.668056 13600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:52:44.675385 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 13600
I20260924 00:52:44.675563 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:52:44.676052 888 external_mini_cluster.cc:1428] Setting key b0e0ae5c475b684da133225b62079678
Sep 24 00:52:44 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
I20260924 00:52:44.713935 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:44.670987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:39469 (local address 127.0.222.62:35035)
0924 00:52:44.686326 (+ 15339us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:44.686333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:44.689274 (+ 2941us) server_negotiation.cc:407] Connection header received
0924 00:52:44.693442 (+ 4168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:44.693449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:44.693571 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:44.693664 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:44.696300 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:44.697503 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:44.700362 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:44.700992 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:44.701097 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:44.705905 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:44.705913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:44.705918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:44.705965 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:44.709498 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:44.710751 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:44.710758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:44.710762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:44.710957 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:44.711563 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:44.711569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:44.711572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:44.711959 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:44.712087 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:44.713177 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0924 00:52:44.713494 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15164,"thread_start_us":15356,"threads_started":1}
I20260924 00:52:44.716315 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:44.671605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:35035 (local address 127.0.222.2:39469)
0924 00:52:44.687063 (+ 15458us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:44.687130 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:44.690772 (+ 3642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:44.694328 (+ 3556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:44.694342 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:44.695011 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:44.696030 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:44.696060 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:44.697904 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:44.697910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:44.700135 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:44.700154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:44.701524 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:44.701531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:44.701837 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:44.702662 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:44.702701 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:44.705566 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:44.709779 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:44.709789 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:44.709811 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:44.710572 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:44.711117 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:44.711122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:44.711126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:44.711416 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:44.712215 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:44.712224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:44.712715 (+ 491us) client_negotiation.cc:769] Sending connection context
0924 00:52:44.714196 (+ 1481us) client_negotiation.cc:240] Negotiation successful
0924 00:52:44.714818 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15080,"thread_start_us":16576,"threads_started":1}
I20260924 00:52:44.718788 13738 heartbeater.cc:347] Connected to a master server at 127.0.222.62:35035
I20260924 00:52:44.719133 13738 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:44.720007 13738 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
I20260924 00:52:44.721944 13379 ts_manager.cc:194] Registered new tserver with Master: 2defc331ddf14257bbeef3cab789a55b (127.0.222.2:44445)
I20260924 00:52:44.723016 13379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:39469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:52:44.749222 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:60711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:35035
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:52:44.828740 13594 heartbeater.cc:503] Master 127.0.222.62:35035 was elected leader, sending a full tablet report...
W20260924 00:52:45.141578 13744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:45.141906 13744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:45.142058 13744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:45.150247 13744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:45.150409 13744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:45.150573 13744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:52:45.161145 13744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.222.20:60711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:45.162575 13744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:45.163887 13744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:45.178119 13750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:45.178524 13749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:45.178731 13752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:45.179428 13744 server_base.cc:1034] running on GCE node
I20260924 00:52:45.180207 13744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:45.181382 13744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:45.182610 13744 hybrid_clock.cc:648] HybridClock initialized: now 1790211165182537 us; error 56 us; skew 500 ppm
I20260924 00:52:45.183038 13744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:45 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:45.186851 13744 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:52:45.188531 13744 webserver.cc:460] Webserver started at http://127.0.222.3:38137/ using document root <none> and password file <none>
I20260924 00:52:45.189554 13744 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:45.189703 13744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:45.190058 13744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:52:45 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:45.207824 13744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "2bd945536f5e423ea59dd6b23ff779f8"
format_stamp: "Formatted at 2026-09-24 00:52:45 on dist-test-slave-rtvz"
server_key: "ff64de31f9b5c1bfa7188ee2dc6589e2"
server_key_iv: "4733c6a451924916776f565c7cb4810c"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:45.208972 13744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "2bd945536f5e423ea59dd6b23ff779f8"
format_stamp: "Formatted at 2026-09-24 00:52:45 on dist-test-slave-rtvz"
server_key: "ff64de31f9b5c1bfa7188ee2dc6589e2"
server_key_iv: "4733c6a451924916776f565c7cb4810c"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:45.216536 13744 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260924 00:52:45.231938 13759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:45.233682 13744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 00:52:45.233892 13744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "2bd945536f5e423ea59dd6b23ff779f8"
format_stamp: "Formatted at 2026-09-24 00:52:45 on dist-test-slave-rtvz"
server_key: "ff64de31f9b5c1bfa7188ee2dc6589e2"
server_key_iv: "4733c6a451924916776f565c7cb4810c"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:45.234131 13744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:45.249614 13744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:45.253499 13744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:45.253861 13744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:45.254886 13744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:52:45.256397 13744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:52:45.256484 13744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:45.256629 13744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:52:45.256699 13744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:45.292553 13744 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:36645
I20260924 00:52:45.292620 13873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:36645 every 8 connection(s)
I20260924 00:52:45.293908 13744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:52:45.299392 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 13744
I20260924 00:52:45.299582 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:52:45.300089 888 external_mini_cluster.cc:1428] Setting key d54ef41bd39feb958d32a4c8f64fa3c8
Sep 24 00:52:45 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:45.316522 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:45.296546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:52559 (local address 127.0.222.62:35035)
0924 00:52:45.297222 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:45.297229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:45.298472 (+ 1243us) server_negotiation.cc:407] Connection header received
0924 00:52:45.300573 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:45.300578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:45.300664 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:45.300803 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:45.302814 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.303678 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.305499 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.306020 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.306115 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:45.310004 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:45.310009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:45.310012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:45.310047 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:45.312878 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:45.313761 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:45.313767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:45.313770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:45.313940 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:45.314460 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:45.314465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:45.314468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:45.314863 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:45.315015 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:45.316022 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0924 00:52:45.316231 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":355,"threads_started":1}
I20260924 00:52:45.317611 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:45.297381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:35035 (local address 127.0.222.3:52559)
0924 00:52:45.298267 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:45.298340 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:45.300260 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:45.301075 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:45.301088 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:45.301750 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:45.302622 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:45.302650 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.303918 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.303922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:45.305367 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:45.305380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.306251 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.306259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:45.306502 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:45.307223 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:45.307249 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:45.309716 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:45.313019 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:45.313026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:45.313040 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:45.313612 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:45.314048 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:45.314057 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:45.314060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:45.314336 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:45.315037 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:45.315045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:45.315491 (+ 446us) client_negotiation.cc:769] Sending connection context
0924 00:52:45.315909 (+ 418us) client_negotiation.cc:240] Negotiation successful
0924 00:52:45.316398 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":368,"threads_started":1}
I20260924 00:52:45.319159 13874 heartbeater.cc:347] Connected to a master server at 127.0.222.62:35035
I20260924 00:52:45.319479 13874 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:45.320340 13874 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
I20260924 00:52:45.321928 13379 ts_manager.cc:194] Registered new tserver with Master: 2bd945536f5e423ea59dd6b23ff779f8 (127.0.222.3:36645)
I20260924 00:52:45.322809 13379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:52559
I20260924 00:52:45.328357 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:52:45.330111 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 13459
Sep 24 00:52:45 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:52:45.348872 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:45.331138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.0.222.1:44719)
0924 00:52:45.331810 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:45.331818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:45.331844 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:52:45.331900 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:45.331907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:45.332117 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:45.332277 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:45.334046 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.335171 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.336780 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.337491 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.337601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:45.342618 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:45.342634 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:45.342651 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:45.342719 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:45.345968 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:45.346704 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:45.346713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:45.346715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:45.346930 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:45.347308 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:45.347311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:45.347313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:45.347713 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:45.347996 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:45.348317 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 00:52:45.348554 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":368,"threads_started":1}
I20260924 00:52:45.350296 13589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:45.433997 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 13459
I20260924 00:52:45.447615 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 13600
Sep 24 00:52:45 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:52:45.464371 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:45.448649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.0.222.2:44445)
0924 00:52:45.449188 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:45.449226 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:45.449257 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:52:45.449338 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:45.449345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:45.449560 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:45.449687 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:45.451084 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.452127 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.453471 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.454005 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.454102 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:45.458388 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:45.458400 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:45.458415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:45.458452 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:45.461178 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:45.462169 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:45.462176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:45.462179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:45.462369 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:45.462822 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:45.462826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:45.462828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:45.463194 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:45.463433 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:45.463723 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 00:52:45.464025 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":284,"threads_started":1}
I20260924 00:52:45.465826 13733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:45.547943 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 13600
I20260924 00:52:45.564662 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 13744
Sep 24 00:52:45 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:52:45.582719 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:45.565785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.0.222.3:36645)
0924 00:52:45.566487 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:45.566494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:45.566527 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:52:45.566587 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:45.566594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:45.566845 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:45.567004 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:45.568548 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.569834 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.571264 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:45.571866 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:45.572010 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:45.576690 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:45.576707 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:45.576723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:45.576771 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:45.579682 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:45.580432 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:45.580440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:45.580443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:45.580583 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:45.580944 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:45.580948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:45.580950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:45.581472 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:45.581789 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:45.582102 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 00:52:45.582369 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":356,"threads_started":1}
I20260924 00:52:45.583971 13869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:45.669492 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 13744
I20260924 00:52:45.682679 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 13346
I20260924 00:52:45.684139 13409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:45.792981 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 13346
I20260924 00:52:45.814491 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:35035
--webserver_interface=127.0.222.62
--webserver_port=41091
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:35035
--ranger_config_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:60711/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:60711/kms with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:52:46.142500 13886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:46.142812 13886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:46.142936 13886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:46.150393 13886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:52:46.150513 13886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:46.150568 13886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:46.150648 13886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:52:46.150708 13886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:52:46.161036 13886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:60711/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:35035
--ranger_config_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=41091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:46.162348 13886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:46.163630 13886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:46.175175 13892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:46.175196 13894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:46.175237 13891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:46.175899 13886 server_base.cc:1034] running on GCE node
I20260924 00:52:46.176759 13886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:46.178237 13886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:46.179468 13886 hybrid_clock.cc:648] HybridClock initialized: now 1790211166179426 us; error 32 us; skew 500 ppm
I20260924 00:52:46.179831 13886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:46 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:46.186596 13886 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:52:46.188282 13886 webserver.cc:460] Webserver started at http://127.0.222.62:41091/ using document root <none> and password file <none>
I20260924 00:52:46.188977 13886 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:46.189090 13886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 00:52:46 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:46.242376 13886 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 00:52:46.245735 13902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:46.247364 13886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:52:46.247584 13886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "5b6ef0863cf94b21950e861711d69ff2"
format_stamp: "Formatted at 2026-09-24 00:52:42 on dist-test-slave-rtvz"
server_key: "5b01f0254222e0e3e4de898243624375"
server_key_iv: "c2a259c45f4b1ba068fd0211708473ba"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:46.248641 13886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:46.265623 13886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:46.269721 13886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:46.269999 13886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:46.287411 13886 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:35035
I20260924 00:52:46.287436 13953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:35035 every 8 connection(s)
I20260924 00:52:46.288982 13886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:52:46.296259 13954 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:52:46.296887 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 13886
I20260924 00:52:46.297983 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:60711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:44719
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:35035
--webserver_port=32883
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:60711/kms with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:52:46.301105 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Bootstrap starting.
I20260924 00:52:46.321588 13954 log.cc:824] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:46.330706 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:52:46.331233 13954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Bootstrap complete.
I20260924 00:52:46.339901 13954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } }
I20260924 00:52:46.340744 13954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6ef0863cf94b21950e861711d69ff2, State: Initialized, Role: FOLLOWER
I20260924 00:52:46.341444 13954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } }
I20260924 00:52:46.341755 13954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:52:46.341882 13954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:52:46.342064 13954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:52:46.346163 13954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } }
I20260924 00:52:46.346719 13954 leader_election.cc:302] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b6ef0863cf94b21950e861711d69ff2; no voters:
I20260924 00:52:46.347383 13954 leader_election.cc:288] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:52:46.347525 13959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:52:46.348699 13959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [term 2 LEADER]: Becoming Leader. State: Replica: 5b6ef0863cf94b21950e861711d69ff2, State: Running, Role: LEADER
I20260924 00:52:46.349418 13959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } }
I20260924 00:52:46.349735 13954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:52:46.352047 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b6ef0863cf94b21950e861711d69ff2. Latest consensus state: current_term: 2 leader_uuid: "5b6ef0863cf94b21950e861711d69ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } } }
I20260924 00:52:46.352443 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:46.353101 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b6ef0863cf94b21950e861711d69ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6ef0863cf94b21950e861711d69ff2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35035 } } }
I20260924 00:52:46.353426 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:46.354719 13967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:52:46.360948 13967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:52:46.362694 13967 catalog_manager.cc:1259] Loaded cluster ID: b8b4e271660343b4b06bca26ff6f8fcb
I20260924 00:52:46.362869 13967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:52:46.367287 13967 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:52:46.369074 13967 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Loaded TSK: 0
I20260924 00:52:46.370790 13967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:52:46.373410 13954 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 00:52:46.710230 13956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:46.710777 13956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:46.710983 13956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:46.722849 13956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:46.723078 13956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:46.723321 13956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:52:46.736366 13956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:60711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:44719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=32883
--tserver_master_addrs=127.0.222.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:46.738224 13956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:46.739910 13956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:46.774842 13992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:46.777585 13993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:46.790549 13956 server_base.cc:1034] running on GCE node
W20260924 00:52:46.791177 13996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:46.791904 13956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:46.793025 13956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:46.794250 13956 hybrid_clock.cc:648] HybridClock initialized: now 1790211166794191 us; error 78 us; skew 500 ppm
I20260924 00:52:46.794585 13956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:46 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:46.798012 13956 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:52:46.808601 13956 webserver.cc:460] Webserver started at http://127.0.222.1:32883/ using document root <none> and password file <none>
I20260924 00:52:46.809574 13956 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:46.809845 13956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 00:52:46 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:46.881811 13956 fs_manager.cc:709] Time spent opening directory manager: real 0.039s user 0.003s sys 0.025s
I20260924 00:52:46.891397 14003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:46.893119 13956 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.006s sys 0.002s
I20260924 00:52:46.893347 13956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "a91ac1681a034147b47af97f6df8a0e6"
format_stamp: "Formatted at 2026-09-24 00:52:43 on dist-test-slave-rtvz"
server_key: "0f1db334d9815f4bda8f925ae0eeb4c6"
server_key_iv: "b68c15be21fb22a1ad5e00988ce2c401"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:46.894428 13956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:46.944319 13956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:46.948665 13956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:46.948966 13956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:46.950205 13956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:52:46.951884 13956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:52:46.951996 13956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:46.952099 13956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:52:46.952167 13956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:47.019201 13956 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:44719
I20260924 00:52:47.019244 14119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:44719 every 8 connection(s)
I20260924 00:52:47.021286 13956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:52:47.024166 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 13956
I20260924 00:52:47.024989 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:60711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:44445
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:35035
--webserver_port=40921
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:60711/kms with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:52:47 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:47.067580 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:47.026019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:35035 (local address 127.0.222.1:47959)
0924 00:52:47.036202 (+ 10183us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:47.036268 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:47.039866 (+ 3598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:47.043095 (+ 3229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:47.043107 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:47.043821 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:47.044764 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:47.044795 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:47.047529 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:47.047535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:47.049987 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:47.050024 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:47.051827 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:47.051838 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:47.052167 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:47.053127 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:47.053170 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:47.056596 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:47.062168 (+ 5572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:47.062179 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:47.062198 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:47.062844 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:47.063441 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:47.063446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:47.063450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:47.063720 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:47.064712 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:47.064721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:47.065134 (+ 413us) client_negotiation.cc:769] Sending connection context
0924 00:52:47.065473 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 00:52:47.066066 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9850,"thread_start_us":8774,"threads_started":1}
I20260924 00:52:47.069157 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:47.025919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:47959 (local address 127.0.222.62:35035)
0924 00:52:47.036332 (+ 10413us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:47.036354 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:47.036423 (+ 69us) server_negotiation.cc:407] Connection header received
0924 00:52:47.041256 (+ 4833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:47.041292 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:47.041858 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:47.042325 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:47.045247 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:47.047278 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:47.050246 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:47.051494 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:47.051736 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:47.057791 (+ 6055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:47.057804 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:47.057830 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:47.057884 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:47.061950 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:47.063070 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:47.063080 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:47.063089 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:47.063292 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:47.063870 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:47.063875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:47.063879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:47.064556 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:47.066616 (+ 2060us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:47.066975 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 00:52:47.067487 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10137,"thread_start_us":9919,"threads_started":1}
I20260924 00:52:47.072958 14120 heartbeater.cc:347] Connected to a master server at 127.0.222.62:35035
I20260924 00:52:47.073365 14120 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:47.074393 14120 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
W20260924 00:52:47.078151 14120 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:35035 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 00:52:47.078445 14120 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:47.078805 14120 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
I20260924 00:52:47.080247 14120 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:47.080570 14120 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
W20260924 00:52:47.082068 14120 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:52:47.410880 14124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:47.411162 14124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:47.411291 14124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:47.418942 14124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:47.419098 14124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:47.419363 14124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:52:47.432632 14124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:60711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:44445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=40921
--tserver_master_addrs=127.0.222.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:47.434042 14124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:47.435313 14124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:47.452651 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:47.452651 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:47.453564 14138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:47.454422 14124 server_base.cc:1034] running on GCE node
I20260924 00:52:47.455309 14124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:47.456650 14124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:47.457942 14124 hybrid_clock.cc:648] HybridClock initialized: now 1790211167457865 us; error 56 us; skew 500 ppm
I20260924 00:52:47.458349 14124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:47 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:47.462667 14124 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:52:47.467496 14124 webserver.cc:460] Webserver started at http://127.0.222.2:40921/ using document root <none> and password file <none>
I20260924 00:52:47.468526 14124 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:47.468703 14124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 00:52:47 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:47.522046 14124 fs_manager.cc:709] Time spent opening directory manager: real 0.031s user 0.012s sys 0.016s
I20260924 00:52:47.533854 14151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:47.538645 14124 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.009s sys 0.001s
I20260924 00:52:47.538910 14124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "2defc331ddf14257bbeef3cab789a55b"
format_stamp: "Formatted at 2026-09-24 00:52:44 on dist-test-slave-rtvz"
server_key: "9aca84766d7142678b190871482dbc52"
server_key_iv: "6b71ddff6949da7b812b54e123175746"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:47.540401 14124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:47.587080 14124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:47.592463 14124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:47.592756 14124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:47.594054 14124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:52:47.595857 14124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:52:47.596004 14124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:47.596141 14124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:52:47.596237 14124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:52:47 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790211167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:47.712141 14124 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:44445
I20260924 00:52:47.712253 14266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:44445 every 8 connection(s)
I20260924 00:52:47.713773 14124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:52:47.722429 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 14124
I20260924 00:52:47.723181 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.222.20:60711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:36645
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:35035
--webserver_port=38137
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:60711/kms with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:52:47 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:47.741843 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:47.716446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:42115 (local address 127.0.222.62:35035)
0924 00:52:47.717650 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:47.717657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:47.718459 (+ 802us) server_negotiation.cc:407] Connection header received
0924 00:52:47.721131 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:47.721138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:47.721274 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:47.721392 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:47.723543 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:47.724743 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:47.727370 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:47.728095 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:47.728205 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:47.733024 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:47.733031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:47.733036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:47.733082 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:47.736648 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:47.737746 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:47.737752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:47.737756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:47.737926 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:47.738743 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:47.738748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:47.738752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:47.739126 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:47.740057 (+ 931us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:47.741140 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0924 00:52:47.741442 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1027,"thread_start_us":915,"threads_started":1}
I20260924 00:52:47.744310 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:47.717356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:35035 (local address 127.0.222.2:42115)
0924 00:52:47.718260 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:47.718312 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:47.720750 (+ 2438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:47.721731 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:47.721744 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:47.722420 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:47.723317 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:47.723345 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:47.724972 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:47.724978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:47.727154 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:47.727185 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:47.728955 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:47.728963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:47.729309 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:47.730078 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:47.730111 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:47.732695 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:47.736846 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:47.736858 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:47.736880 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:47.737584 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:47.738290 (+ 706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:47.738296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:47.738300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:47.738593 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:47.740169 (+ 1576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:47.740179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:47.740658 (+ 479us) client_negotiation.cc:769] Sending connection context
0924 00:52:47.742173 (+ 1515us) client_negotiation.cc:240] Negotiation successful
0924 00:52:47.742811 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":365,"threads_started":1}
I20260924 00:52:47.746362 14267 heartbeater.cc:347] Connected to a master server at 127.0.222.62:35035
I20260924 00:52:47.746675 14267 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:47.747560 14267 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
I20260924 00:52:47.749751 13919 ts_manager.cc:194] Registered new tserver with Master: 2defc331ddf14257bbeef3cab789a55b (127.0.222.2:44445)
I20260924 00:52:47.751986 13919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:42115
Sep 24 00:52:48 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:48.082257 14120 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:48.082736 14120 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
I20260924 00:52:48.084012 13919 ts_manager.cc:194] Registered new tserver with Master: a91ac1681a034147b47af97f6df8a0e6 (127.0.222.1:44719)
I20260924 00:52:48.085191 13919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:47959
W20260924 00:52:48.236644 14271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:48.237175 14271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:48.237433 14271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:48.249488 14271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:48.249732 14271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:48.249977 14271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:52:48.266163 14271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.222.20:60711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:36645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=38137
--tserver_master_addrs=127.0.222.62:35035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:48.268049 14271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:48.270092 14271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:48.287627 14276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:48.298657 14277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:48.301162 14279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:48.302608 14271 server_base.cc:1034] running on GCE node
I20260924 00:52:48.303511 14271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:48.304932 14271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:48.306216 14271 hybrid_clock.cc:648] HybridClock initialized: now 1790211168306091 us; error 97 us; skew 500 ppm
I20260924 00:52:48.306638 14271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:48 dist-test-slave-rtvz krb5kdc[12898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:48.311165 14271 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:52:48.313285 14271 webserver.cc:460] Webserver started at http://127.0.222.3:38137/ using document root <none> and password file <none>
I20260924 00:52:48.314236 14271 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:48.314399 14271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 00:52:48 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:48.348500 14271 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.004s
I20260924 00:52:48.356858 14287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:48.359141 14271 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260924 00:52:48.359489 14271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "2bd945536f5e423ea59dd6b23ff779f8"
format_stamp: "Formatted at 2026-09-24 00:52:45 on dist-test-slave-rtvz"
server_key: "ff64de31f9b5c1bfa7188ee2dc6589e2"
server_key_iv: "4733c6a451924916776f565c7cb4810c"
server_key_version: "kuduclusterkey@0"
I20260924 00:52:48.361047 14271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 24 00:52:48 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for HTTP/127.0.222.20@KRBTEST.COM
I20260924 00:52:48.403072 14271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:48.408463 14271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:48.408849 14271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:48.424495 14271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:52:48.446071 14271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:52:48.446327 14271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 00:52:48.446530 14271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:52:48.446671 14271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:48.601872 14271 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:36645
I20260924 00:52:48.601914 14401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:36645 every 8 connection(s)
I20260924 00:52:48.603454 14271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:52:48.610528 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 14271
Sep 24 00:52:48 dist-test-slave-rtvz krb5kdc[12898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:48.637894 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.607837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:48827 (local address 127.0.222.62:35035)
0924 00:52:48.613234 (+ 5397us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.613241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.613262 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:52:48.613339 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.613344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.613460 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.613575 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:48.618340 (+ 4765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.619472 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.621944 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.622368 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.622474 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.628128 (+ 5654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:48.628135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:48.628140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:48.628215 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:48.632667 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:48.633704 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:48.633710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:48.633714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:48.633870 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:48.634417 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:48.634422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:48.634425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:48.634763 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:48.634898 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.637322 (+ 2424us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.637552 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5236,"thread_start_us":5684,"threads_started":1}
I20260924 00:52:48.638317 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.607177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:35035 (local address 127.0.222.3:48827)
0924 00:52:48.609587 (+ 2410us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:48.609636 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:48.612065 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:48.616929 (+ 4864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:48.616941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:48.617521 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:48.618152 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.618171 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.619677 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.619683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.621790 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.621814 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.622808 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.622816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.623023 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.625227 (+ 2204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:48.625263 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:48.627815 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:48.632847 (+ 5032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:48.632857 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:48.632876 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:48.633545 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:48.633999 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:48.634004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:48.634008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:48.634271 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:48.634977 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:48.634985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:48.635447 (+ 462us) client_negotiation.cc:769] Sending connection context
0924 00:52:48.635747 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 00:52:48.636327 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2108,"thread_start_us":1897,"threads_started":1}
I20260924 00:52:48.640281 14402 heartbeater.cc:347] Connected to a master server at 127.0.222.62:35035
I20260924 00:52:48.640705 14402 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:48.642088 14402 heartbeater.cc:511] Master 127.0.222.62:35035 requested a full tablet report, sending...
I20260924 00:52:48.643900 13919 ts_manager.cc:194] Registered new tserver with Master: 2bd945536f5e423ea59dd6b23ff779f8 (127.0.222.3:36645)
I20260924 00:52:48.644979 13919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:48827
I20260924 00:52:48.645807 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.611581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.0.222.62:35035)
0924 00:52:48.616016 (+ 4435us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.616021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.616682 (+ 661us) server_negotiation.cc:407] Connection header received
0924 00:52:48.616769 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.616774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.616847 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.616948 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:48.620403 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.621617 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.623629 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.624297 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.624438 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.628081 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:48.628089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:48.628093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:48.628144 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:48.639802 (+ 11658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:48.642313 (+ 2511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:48.642319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:48.642323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:48.642518 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:48.643087 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:48.643093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:48.643096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:48.643463 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:48.644993 (+ 1530us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.645124 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.645405 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4306,"thread_start_us":4159,"threads_started":1}
I20260924 00:52:48.649653 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:52:48.671332 14407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.655208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.0.222.62:35035)
0924 00:52:48.655405 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.655412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.656023 (+ 611us) server_negotiation.cc:407] Connection header received
0924 00:52:48.656165 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.656171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.656279 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.656404 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:48.658188 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.659280 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.660663 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.661406 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.661540 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.664790 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:48.664797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:48.664803 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:48.664850 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:48.668366 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:48.669162 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:48.669168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:48.669172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:48.669372 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:48.669769 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:48.669775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:48.669778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:48.670182 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:48.670425 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.670726 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.670983 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:52:48.677637 13919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:52:48.688853 13919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:52:48.733186 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.704931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.0.1:35120)
0924 00:52:48.720445 (+ 15514us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:48.720500 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:48.720793 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:48.721935 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:48.721947 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:48.721979 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:52:48.722385 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.722410 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.725875 (+ 3465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.725881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.729758 (+ 3877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.729779 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.732074 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.732081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.732287 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.732341 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 00:52:48.732491 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 00:52:48.732728 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14449,"mutex_wait_us":1,"thread_start_us":14643,"threads_started":1}
I20260924 00:52:48.734301 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.705722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.0.222.2:44445)
0924 00:52:48.723630 (+ 17908us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.723639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.724356 (+ 717us) server_negotiation.cc:407] Connection header received
0924 00:52:48.724460 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.724468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.724722 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.724947 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:52:48.725825 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.727360 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.731320 (+ 3960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.732694 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.732787 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.732995 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.733820 (+ 825us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.733984 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17739,"thread_start_us":23921,"threads_started":1}
I20260924 00:52:48.735256 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.705184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.222.1:44719)
0924 00:52:48.718832 (+ 13648us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.718842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.720635 (+ 1793us) server_negotiation.cc:407] Connection header received
0924 00:52:48.721063 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.721071 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.721372 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.721582 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:52:48.722609 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.725539 (+ 2930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.729963 (+ 4424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.731570 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.731698 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.731933 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.733474 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.733648 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13397,"thread_start_us":13633,"threads_started":1}
I20260924 00:52:48.735311 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.705292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:44445 (local address 127.0.0.1:55412)
0924 00:52:48.724036 (+ 18744us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:48.724058 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:48.724228 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:48.725182 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:48.725188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:48.725236 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:52:48.725598 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.725612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.727542 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.727548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.731134 (+ 3586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.731149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.733247 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.733254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.733616 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.733647 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 00:52:48.734735 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0924 00:52:48.734895 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18627,"mutex_wait_us":1433,"thread_start_us":24563,"threads_started":1}
I20260924 00:52:48.739693 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.722027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36645 (local address 127.0.0.1:43488)
0924 00:52:48.731036 (+ 9009us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:48.731056 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:48.731227 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:48.732277 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:48.732282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:48.732306 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:52:48.732676 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.732689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.734688 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.734693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.736746 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.736758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.738951 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.738958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.739128 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.739160 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:52:48.739247 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:52:48.739372 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8871,"thread_start_us":8685,"threads_started":1}
I20260924 00:52:48.739745 14054 tablet_service.cc:1467] Processing CreateTablet for tablet 4099b3484148481ba032efef538bc658 (DEFAULT_TABLE table=test-table [id=3d0dc2a45d5c41139282a5e25d030d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:52:48.740411 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.722783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.222.3:36645)
0924 00:52:48.730563 (+ 7780us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.730572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.731361 (+ 789us) server_negotiation.cc:407] Connection header received
0924 00:52:48.731466 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.731474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.731762 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.731982 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:52:48.732884 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.734526 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.736899 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.738356 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.738487 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.738724 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.739930 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.740096 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7588,"thread_start_us":7413,"threads_started":1}
I20260924 00:52:48.740641 14200 tablet_service.cc:1467] Processing CreateTablet for tablet 4099b3484148481ba032efef538bc658 (DEFAULT_TABLE table=test-table [id=3d0dc2a45d5c41139282a5e25d030d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:52:48.741353 14054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4099b3484148481ba032efef538bc658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:52:48.742537 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4099b3484148481ba032efef538bc658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:52:48.748950 14336 tablet_service.cc:1467] Processing CreateTablet for tablet 4099b3484148481ba032efef538bc658 (DEFAULT_TABLE table=test-table [id=3d0dc2a45d5c41139282a5e25d030d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:52:48.750972 14336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4099b3484148481ba032efef538bc658. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:52:48.759995 14267 heartbeater.cc:503] Master 127.0.222.62:35035 was elected leader, sending a full tablet report...
I20260924 00:52:48.766469 14430 tablet_bootstrap.cc:492] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6: Bootstrap starting.
I20260924 00:52:48.772549 14430 tablet_bootstrap.cc:654] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:48.773448 14431 tablet_bootstrap.cc:492] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: Bootstrap starting.
I20260924 00:52:48.777045 14430 log.cc:824] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:48.778021 14431 tablet_bootstrap.cc:654] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:48.779278 14431 log.cc:824] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:48.782095 14432 tablet_bootstrap.cc:492] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Bootstrap starting.
I20260924 00:52:48.786264 14431 tablet_bootstrap.cc:492] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: No bootstrap required, opened a new log
I20260924 00:52:48.786264 14430 tablet_bootstrap.cc:492] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6: No bootstrap required, opened a new log
I20260924 00:52:48.786274 14432 tablet_bootstrap.cc:654] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:48.786598 14430 ts_tablet_manager.cc:1397] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.012s
I20260924 00:52:48.786657 14431 ts_tablet_manager.cc:1397] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260924 00:52:48.787609 14432 log.cc:824] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:48.797514 14430 raft_consensus.cc:348] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.797514 14431 raft_consensus.cc:348] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.798025 14431 raft_consensus.cc:374] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:48.798081 14430 raft_consensus.cc:374] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:48.798216 14430 raft_consensus.cc:729] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91ac1681a034147b47af97f6df8a0e6, State: Initialized, Role: FOLLOWER
I20260924 00:52:48.798960 14430 consensus_queue.cc:260] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.799225 14431 raft_consensus.cc:729] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2defc331ddf14257bbeef3cab789a55b, State: Initialized, Role: FOLLOWER
I20260924 00:52:48.800043 14431 consensus_queue.cc:260] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.801761 14432 tablet_bootstrap.cc:492] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: No bootstrap required, opened a new log
I20260924 00:52:48.802134 14432 ts_tablet_manager.cc:1397] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.013s
I20260924 00:52:48.805387 14431 ts_tablet_manager.cc:1428] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: Time spent starting tablet: real 0.019s user 0.013s sys 0.003s
I20260924 00:52:48.812810 14432 raft_consensus.cc:348] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.813438 14120 heartbeater.cc:503] Master 127.0.222.62:35035 was elected leader, sending a full tablet report...
I20260924 00:52:48.813513 14432 raft_consensus.cc:374] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:48.813663 14432 raft_consensus.cc:729] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd945536f5e423ea59dd6b23ff779f8, State: Initialized, Role: FOLLOWER
I20260924 00:52:48.814438 14432 consensus_queue.cc:260] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.815645 14430 ts_tablet_manager.cc:1428] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6: Time spent starting tablet: real 0.029s user 0.016s sys 0.011s
I20260924 00:52:48.817139 14402 heartbeater.cc:503] Master 127.0.222.62:35035 was elected leader, sending a full tablet report...
I20260924 00:52:48.817245 14432 ts_tablet_manager.cc:1428] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
I20260924 00:52:48.828909 14437 raft_consensus.cc:482] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:52:48.829373 14437 raft_consensus.cc:504] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.831372 14437 leader_election.cc:288] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91ac1681a034147b47af97f6df8a0e6 (127.0.222.1:44719), 2bd945536f5e423ea59dd6b23ff779f8 (127.0.222.3:36645)
I20260924 00:52:48.838923 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.832183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:43939 (local address 127.0.222.3:36645)
0924 00:52:48.832382 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.832388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.832951 (+ 563us) server_negotiation.cc:407] Connection header received
0924 00:52:48.833080 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.833086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.833161 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.833279 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:52:48.833964 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.834989 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.837314 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.838058 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.838169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.838282 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.838556 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.838687 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:52:48.839084 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.831691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36645 (local address 127.0.222.2:43939)
0924 00:52:48.832823 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:48.832841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:48.832976 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:48.833456 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:48.833461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:48.833486 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:52:48.833809 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.833819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.835144 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.835149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.837172 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.837183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.838246 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.838253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.838412 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.838435 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 00:52:48.838516 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:52:48.838639 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":24,"thread_start_us":824,"threads_started":1}
I20260924 00:52:48.839613 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.831968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:55213 (local address 127.0.222.1:44719)
0924 00:52:48.832172 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.832178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.832295 (+ 117us) server_negotiation.cc:407] Connection header received
0924 00:52:48.832614 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.832620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.832711 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.832777 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:52:48.833531 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.834618 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.837235 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.838455 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.838574 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.838670 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.839240 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.839379 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:52:48.840092 14357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4099b3484148481ba032efef538bc658" candidate_uuid: "2defc331ddf14257bbeef3cab789a55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd945536f5e423ea59dd6b23ff779f8" is_pre_election: true
I20260924 00:52:48.840415 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.831550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.222.2:55213)
0924 00:52:48.832158 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:48.832177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:48.832421 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:48.832978 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:48.832983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:48.833002 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:52:48.833363 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.833376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.834793 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.834798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.837041 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:48.837056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.838840 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.838847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:48.839045 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.839083 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:52:48.839911 (+ 828us) client_negotiation.cc:240] Negotiation successful
0924 00:52:48.840051 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":3,"thread_start_us":627,"threads_started":1}
I20260924 00:52:48.840618 14357 raft_consensus.cc:2454] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2defc331ddf14257bbeef3cab789a55b in term 0.
I20260924 00:52:48.841377 14153 leader_election.cc:302] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bd945536f5e423ea59dd6b23ff779f8, 2defc331ddf14257bbeef3cab789a55b; no voters:
I20260924 00:52:48.841810 14437 raft_consensus.cc:2781] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:52:48.841967 14437 raft_consensus.cc:482] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:52:48.842054 14437 raft_consensus.cc:3037] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:52:48.841832 14075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4099b3484148481ba032efef538bc658" candidate_uuid: "2defc331ddf14257bbeef3cab789a55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91ac1681a034147b47af97f6df8a0e6" is_pre_election: true
I20260924 00:52:48.842396 14075 raft_consensus.cc:2454] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2defc331ddf14257bbeef3cab789a55b in term 0.
I20260924 00:52:48.844518 14437 raft_consensus.cc:504] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.845084 14437 leader_election.cc:288] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [CANDIDATE]: Term 1 election: Requested vote from peers a91ac1681a034147b47af97f6df8a0e6 (127.0.222.1:44719), 2bd945536f5e423ea59dd6b23ff779f8 (127.0.222.3:36645)
I20260924 00:52:48.845541 14075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4099b3484148481ba032efef538bc658" candidate_uuid: "2defc331ddf14257bbeef3cab789a55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91ac1681a034147b47af97f6df8a0e6"
I20260924 00:52:48.845634 14357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4099b3484148481ba032efef538bc658" candidate_uuid: "2defc331ddf14257bbeef3cab789a55b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd945536f5e423ea59dd6b23ff779f8"
I20260924 00:52:48.845757 14075 raft_consensus.cc:3037] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:52:48.845855 14357 raft_consensus.cc:3037] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:52:48.849025 14357 raft_consensus.cc:2454] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2defc331ddf14257bbeef3cab789a55b in term 1.
I20260924 00:52:48.849359 14075 raft_consensus.cc:2454] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2defc331ddf14257bbeef3cab789a55b in term 1.
I20260924 00:52:48.849557 14153 leader_election.cc:302] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bd945536f5e423ea59dd6b23ff779f8, 2defc331ddf14257bbeef3cab789a55b; no voters:
I20260924 00:52:48.849994 14437 raft_consensus.cc:2781] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:52:48.850560 14437 raft_consensus.cc:686] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 1 LEADER]: Becoming Leader. State: Replica: 2defc331ddf14257bbeef3cab789a55b, State: Running, Role: LEADER
I20260924 00:52:48.851121 14437 consensus_queue.cc:237] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } }
I20260924 00:52:48.856353 13918 catalog_manager.cc:5495] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2defc331ddf14257bbeef3cab789a55b (127.0.222.2). New cstate: current_term: 1 leader_uuid: "2defc331ddf14257bbeef3cab789a55b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2defc331ddf14257bbeef3cab789a55b" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 44445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 } health_report { overall_health: UNKNOWN } } }
W20260924 00:52:48.865577 14403 tablet.cc:2369] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:52:48.879240 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.873574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.0.222.2:44445)
0924 00:52:48.874128 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.874132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.874147 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:52:48.874231 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.874235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.874327 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.874465 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:52:48.874870 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.875608 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.877031 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.877498 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.877608 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.878132 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:52:48.878662 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:52:48.878809 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.878889 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.878977 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426}
I20260924 00:52:48.887735 14357 raft_consensus.cc:1261] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 1 FOLLOWER]: Refusing update from remote peer 2defc331ddf14257bbeef3cab789a55b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:52:48.887857 14075 raft_consensus.cc:1261] T 4099b3484148481ba032efef538bc658 P a91ac1681a034147b47af97f6df8a0e6 [term 1 FOLLOWER]: Refusing update from remote peer 2defc331ddf14257bbeef3cab789a55b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:52:48.888795 14437 consensus_queue.cc:1035] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bd945536f5e423ea59dd6b23ff779f8" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:52:48.889076 14441 consensus_queue.cc:1035] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91ac1681a034147b47af97f6df8a0e6" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 44719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:52:48.934819 13918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:52:48.935199 13918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:52:48.940037 13918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b6ef0863cf94b21950e861711d69ff2: Sending DeleteTablet for 3 replicas of tablet 4099b3484148481ba032efef538bc658
I20260924 00:52:48.941524 14200 tablet_service.cc:1514] Processing DeleteTablet for tablet 4099b3484148481ba032efef538bc658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:52:48 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:55412
I20260924 00:52:48.941537 14054 tablet_service.cc:1514] Processing DeleteTablet for tablet 4099b3484148481ba032efef538bc658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:52:48 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:35120
I20260924 00:52:48.943313 14336 tablet_service.cc:1514] Processing DeleteTablet for tablet 4099b3484148481ba032efef538bc658 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:52:48 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:43488
I20260924 00:52:48.950443 14456 tablet_replica.cc:330] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: stopping tablet replica
I20260924 00:52:48.951886 14456 raft_consensus.cc:2229] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:52:48.953330 14456 raft_consensus.cc:2258] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:52:48.954180 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 13956
I20260924 00:52:48.958361 14456 ts_tablet_manager.cc:1905] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:52:48.981467 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:48.955477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.0.222.1:44719)
0924 00:52:48.956153 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:48.956159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:48.956189 (+ 30us) server_negotiation.cc:407] Connection header received
0924 00:52:48.956256 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:48.956260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:48.956341 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:48.956452 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:48.964162 (+ 7710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.965517 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.967569 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:48.968363 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:48.968495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:48.972282 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:48.972306 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:48.972329 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:48.972383 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:48.976468 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:48.977561 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:48.977574 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:48.977578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:48.977817 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:48.978256 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:48.978262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:48.978266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:48.978865 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:48.979216 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:48.980679 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0924 00:52:48.981000 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534}
I20260924 00:52:48.982249 14115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:48.988628 14456 ts_tablet_manager.cc:1918] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:52:48.990413 14456 log.cc:1196] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/4099b3484148481ba032efef538bc658
I20260924 00:52:48.991436 14456 ts_tablet_manager.cc:1939] T 4099b3484148481ba032efef538bc658 P 2bd945536f5e423ea59dd6b23ff779f8: Deleting consensus metadata
I20260924 00:52:48.995524 13903 catalog_manager.cc:4841] TS 2bd945536f5e423ea59dd6b23ff779f8 (127.0.222.3:36645): tablet 4099b3484148481ba032efef538bc658 (table test-table [id=3d0dc2a45d5c41139282a5e25d030d73]) successfully deleted
I20260924 00:52:48.999328 14454 tablet_replica.cc:330] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: stopping tablet replica
I20260924 00:52:49.000425 14454 raft_consensus.cc:2229] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:52:49.001516 14454 raft_consensus.cc:2258] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:52:49.007401 14454 ts_tablet_manager.cc:1905] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:52:49.016402 14454 ts_tablet_manager.cc:1918] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:52:49.016722 14454 log.cc:1196] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/4099b3484148481ba032efef538bc658
I20260924 00:52:49.017416 14454 ts_tablet_manager.cc:1939] T 4099b3484148481ba032efef538bc658 P 2defc331ddf14257bbeef3cab789a55b: Deleting consensus metadata
I20260924 00:52:49.019491 13904 catalog_manager.cc:4841] TS 2defc331ddf14257bbeef3cab789a55b (127.0.222.2:44445): tablet 4099b3484148481ba032efef538bc658 (table test-table [id=3d0dc2a45d5c41139282a5e25d030d73]) successfully deleted
I20260924 00:52:49.095472 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 13956
I20260924 00:52:49.109440 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 14124
W20260924 00:52:49.109283 13906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:44719 (error 108)
I20260924 00:52:49.111804 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.111272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.0.1:35132)
0924 00:52:49.111521 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:49.111633 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260924 00:52:49.112216 13906 catalog_manager.cc:4584] TS a91ac1681a034147b47af97f6df8a0e6 (127.0.222.1:44719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4099b3484148481ba032efef538bc658: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44719: connect: Connection refused (error 111)
I20260924 00:52:49.124436 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.110807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.0.222.2:44445)
0924 00:52:49.111002 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:49.111007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:49.111025 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:52:49.111090 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:49.111095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:49.111182 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:49.111285 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:49.113149 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:49.114043 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:49.115346 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:49.115780 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:49.115884 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:49.118403 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:49.118418 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:49.118435 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:49.118475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:49.121725 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:49.122409 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:49.122416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:49.122419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:49.122571 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:49.122934 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:49.122938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:49.122940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:49.123288 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:49.123588 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:49.123887 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 00:52:49.124115 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 00:52:49.125152 14262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:49.166090 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.165541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.0.1:35134)
0924 00:52:49.165820 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:49.165932 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 00:52:49.167119 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.166749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.0.1:35140)
0924 00:52:49.166904 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:49.166986 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 00:52:49.227833 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.227169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.0.1:35148)
0924 00:52:49.227453 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:49.227550 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 00:52:49.229010 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.228569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.0.1:35160)
0924 00:52:49.228741 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:49.228882 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 00:52:49.234241 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 14124
I20260924 00:52:49.249287 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 14271
I20260924 00:52:49.263077 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.250013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.222.3:36645)
0924 00:52:49.250198 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:49.250202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:49.250221 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:52:49.250435 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:49.250439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:49.250527 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:49.250625 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:49.251997 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:49.252840 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:49.254210 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:49.254608 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:49.254871 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:49.257406 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:49.257421 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:49.257438 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:49.257474 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:49.260249 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:49.261002 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:49.261010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:49.261012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:49.261144 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:49.261547 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:49.261552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:49.261555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:49.261932 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:49.262207 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:49.262518 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 00:52:49.262753 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:52:49.263738 14397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:49.309520 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.309058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.0.1:35170)
0924 00:52:49.309283 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:49.309374 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 00:52:49.310624 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:49.310219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:44719 (local address 127.0.0.1:35178)
0924 00:52:49.310418 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:49.310503 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 00:52:49.353691 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 14271
I20260924 00:52:49.367697 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 13886
I20260924 00:52:49.369086 13949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:49.481129 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 13886
I20260924 00:52:49.503301 888 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 00:52:49.839099 888 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 00:52:49.839346 888 mini_ranger.cc:66] Stopping Ranger...
I20260924 00:52:50.185448 888 mini_ranger.cc:68] Stopped Ranger
2026-09-24 00:52:50.185 UTC [12928] LOG: received smart shutdown request
2026-09-24 00:52:50.189 UTC [12928] LOG: background worker "logical replication launcher" (PID 12938) exited with exit code 1
2026-09-24 00:52:50.189 UTC [12932] LOG: shutting down
2026-09-24 00:52:50.352 UTC [12928] LOG: database system is shut down
2026-09-24T00:52:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54175 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:52:50 dist-test-slave-rtvz krb5kdc[14470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:52:50 dist-test-slave-rtvz krb5kdc[14470](info): set up 2 sockets
Sep 24 00:52:50 dist-test-slave-rtvz krb5kdc[14470](info): commencing operation
krb5kdc: starting...
W20260924 00:52:51.488703 888 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:52:51 dist-test-slave-rtvz krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:52:51.713160 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42429
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42429,127.0.222.61:39499,127.0.222.60:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:52:52.048458 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:52.048775 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:52.048893 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:52.056370 14483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:52:52.056490 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:52.056548 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:52.056627 14483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:52:52.056680 14483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:52:52.066874 14483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42429,127.0.222.61:39499,127.0.222.60:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:52.068307 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:52.069890 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:52.081709 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:52.081693 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:52.081743 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:52.082731 14483 server_base.cc:1034] running on GCE node
I20260924 00:52:52.083563 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:52.085086 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:52.086318 14483 hybrid_clock.cc:648] HybridClock initialized: now 1790211172086278 us; error 37 us; skew 500 ppm
I20260924 00:52:52.086668 14483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:52 dist-test-slave-rtvz krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:52.093454 14483 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:52:52.095008 14483 webserver.cc:460] Webserver started at http://127.0.222.62:37055/ using document root <none> and password file <none>
I20260924 00:52:52.095719 14483 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:52.095832 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:52.096118 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:52:52.099243 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "67a209e67f9744eb8d2746fe481de2c2"
format_stamp: "Formatted at 2026-09-24 00:52:52 on dist-test-slave-rtvz"
server_key: "00dd02ebc1a31ac8418282b16ebbb32b"
server_key_iv: "eb39e3d665ac1c9e684250f41011c8bc"
server_key_version: "encryptionkey@0"
I20260924 00:52:52.100035 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "67a209e67f9744eb8d2746fe481de2c2"
format_stamp: "Formatted at 2026-09-24 00:52:52 on dist-test-slave-rtvz"
server_key: "00dd02ebc1a31ac8418282b16ebbb32b"
server_key_iv: "eb39e3d665ac1c9e684250f41011c8bc"
server_key_version: "encryptionkey@0"
I20260924 00:52:52.105170 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 00:52:52.108629 14499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:52.110046 14483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:52:52.110203 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "67a209e67f9744eb8d2746fe481de2c2"
format_stamp: "Formatted at 2026-09-24 00:52:52 on dist-test-slave-rtvz"
server_key: "00dd02ebc1a31ac8418282b16ebbb32b"
server_key_iv: "eb39e3d665ac1c9e684250f41011c8bc"
server_key_version: "encryptionkey@0"
I20260924 00:52:52.110409 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:52.133925 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:52.138386 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:52.138656 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:52.156334 14483 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42429
I20260924 00:52:52.156374 14550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42429 every 8 connection(s)
I20260924 00:52:52.157773 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:52:52.161563 14551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:52:52.165513 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 14483
I20260924 00:52:52.165740 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:52:52.166508 888 external_mini_cluster.cc:1428] Setting key 2af728c1eb8930e26ba8a89b44919901
I20260924 00:52:52.169471 14551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:52:52 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.61@KRBTEST.COM" created.
I20260924 00:52:52.192327 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.174342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.0.222.62:42429)
0924 00:52:52.174980 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:52.174987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:52.175046 (+ 59us) server_negotiation.cc:407] Connection header received
0924 00:52:52.176448 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:52.176454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:52.176686 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:52.176870 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:52.178128 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.179340 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.180753 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.181358 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.181528 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:52.185487 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:52.185495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:52.185509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:52.185542 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:52.188209 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:52.189037 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:52.189044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:52.189046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:52.189232 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:52.189652 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:52.189655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:52.189657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:52.190173 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:52.190329 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:52.190948 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 00:52:52.191224 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":445,"thread_start_us":288,"threads_started":1}
I20260924 00:52:52.192389 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.173179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42429 (local address 127.0.0.1:49174)
0924 00:52:52.174068 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.174117 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:52.176088 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:52.176906 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:52.176912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:52.177607 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:52.177948 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:52.177965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.179484 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.179489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:52.180613 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:52.180626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.181579 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.181587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:52.181868 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:52.182600 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:52.182634 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:52.185265 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:52.188332 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:52.188344 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:52.188358 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:52.188876 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:52.189346 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:52.189351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:52.189354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:52.189553 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:52.190309 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:52.190318 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:52.190549 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 00:52:52.190817 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 00:52:52.191378 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":109,"thread_start_us":347,"threads_started":1}
I20260924 00:52:52.195657 14551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:52:52.196797 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.196352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41882)
0924 00:52:52.196525 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.196652 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 00:52:52.197129 14500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
I20260924 00:52:52.198812 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.198418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41896)
0924 00:52:52.198571 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.198650 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 00:52:52.199000 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:39499: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab.
I20260924 00:52:52.216174 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:39499
--webserver_interface=127.0.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42429,127.0.222.61:39499,127.0.222.60:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:52:52.248404 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } attempt: 1
I20260924 00:52:52.249752 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.249148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41910)
0924 00:52:52.249435 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.249568 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260924 00:52:52.251339 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.250957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41922)
0924 00:52:52.251120 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.251197 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 00:52:52.251560 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:39499: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
I20260924 00:52:52.319900 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } attempt: 2
I20260924 00:52:52.321115 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.320600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41932)
0924 00:52:52.320868 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.320966 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 00:52:52.322324 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.321924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41940)
0924 00:52:52.322093 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.322177 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 00:52:52.322590 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:39499: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
I20260924 00:52:52.413901 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } attempt: 3
I20260924 00:52:52.415112 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.414609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41954)
0924 00:52:52.414809 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.414970 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 00:52:52.416282 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.415875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41970)
0924 00:52:52.416041 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.416121 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 00:52:52.416514 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:39499: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
W20260924 00:52:52.539321 14558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:52.539608 14558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:52.539743 14558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:52.547061 14558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:52:52.547180 14558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:52.547232 14558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:52.547312 14558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:52:52.547367 14558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:52:52.557552 14558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42429,127.0.222.61:39499,127.0.222.60:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:52.558985 14558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:52.559819 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } attempt: 4
I20260924 00:52:52.560388 14558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:52:52.561023 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.560487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41978)
0924 00:52:52.560802 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.560894 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260924 00:52:52.562131 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.561699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:41988)
0924 00:52:52.561893 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.561991 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260924 00:52:52.562358 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:39499: Network error: Client connection negotiation failed: client connection to 127.0.222.61:39499: connect: Connection refused (error 111)
W20260924 00:52:52.572973 14564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:52.572995 14566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:52.572929 14563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:52.578190 14558 server_base.cc:1034] running on GCE node
I20260924 00:52:52.579077 14558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:52.580900 14558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:52.582130 14558 hybrid_clock.cc:648] HybridClock initialized: now 1790211172582075 us; error 49 us; skew 500 ppm
I20260924 00:52:52.582491 14558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:52 dist-test-slave-rtvz krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:52.585950 14558 init.cc:377] Logged in from keytab as kudu/127.0.222.61@KRBTEST.COM (short username kudu)
I20260924 00:52:52.587604 14558 webserver.cc:460] Webserver started at http://127.0.222.61:46679/ using document root <none> and password file <none>
I20260924 00:52:52.588366 14558 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:52.588480 14558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:52.588763 14558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:52:52.592015 14558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/data/instance:
uuid: "691fb3656720438b95b649bd05b30552"
format_stamp: "Formatted at 2026-09-24 00:52:52 on dist-test-slave-rtvz"
server_key: "a743878dbf5c9c68049acfd317988d9f"
server_key_iv: "737a85b754894ecfe09caf329c05758d"
server_key_version: "encryptionkey@0"
I20260924 00:52:52.592796 14558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/wal/instance:
uuid: "691fb3656720438b95b649bd05b30552"
format_stamp: "Formatted at 2026-09-24 00:52:52 on dist-test-slave-rtvz"
server_key: "a743878dbf5c9c68049acfd317988d9f"
server_key_iv: "737a85b754894ecfe09caf329c05758d"
server_key_version: "encryptionkey@0"
I20260924 00:52:52.598374 14558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 00:52:52.602073 14573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:52.603680 14558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:52:52.603865 14558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/wal
uuid: "691fb3656720438b95b649bd05b30552"
format_stamp: "Formatted at 2026-09-24 00:52:52 on dist-test-slave-rtvz"
server_key: "a743878dbf5c9c68049acfd317988d9f"
server_key_iv: "737a85b754894ecfe09caf329c05758d"
server_key_version: "encryptionkey@0"
I20260924 00:52:52.604074 14558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:52.625447 14558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:52.629741 14558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:52.630026 14558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:52.647632 14558 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:39499
I20260924 00:52:52.647662 14624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:39499 every 8 connection(s)
I20260924 00:52:52.649025 14558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
I20260924 00:52:52.652643 14625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:52:52.658032 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 14558
I20260924 00:52:52.658205 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-1/wal/instance
I20260924 00:52:52.658627 888 external_mini_cluster.cc:1428] Setting key 8d69ada79576b6422eb0e5f93db2a7b5
I20260924 00:52:52.659263 14625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:52:52 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.60@KRBTEST.COM; defaulting to no policy
I20260924 00:52:52.681325 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.662015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.0.222.62:42429)
0924 00:52:52.662200 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:52.662205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:52.663708 (+ 1503us) server_negotiation.cc:407] Connection header received
0924 00:52:52.665644 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:52.665651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:52.665735 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:52.665822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:52.667681 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.668377 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.670541 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.671000 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.671100 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:52.674976 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:52.674982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:52.674985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:52.675019 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:52.677825 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:52.678888 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:52.678892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:52.678894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:52.679035 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:52.679513 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:52.679516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:52.679518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:52.679871 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:52.679992 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:52.680796 (+ 804us) server_negotiation.cc:299] Negotiation successful
0924 00:52:52.681000 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.60@KRBTEST.COM" created.
I20260924 00:52:52.682376 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.662679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42429 (local address 127.0.0.1:49188)
0924 00:52:52.663522 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.663562 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:52.665358 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:52.666072 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:52.666081 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:52.666746 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:52.667487 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:52.667512 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.668535 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.668540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:52.670387 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:52.670403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.671217 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.671224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:52.671496 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:52.672213 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:52.672241 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:52.674727 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:52.677993 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:52.678003 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:52.678023 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:52.678671 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:52.679153 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:52.679158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:52.679162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:52.679400 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:52.679990 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:52.679997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:52.680368 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 00:52:52.680654 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 00:52:52.681071 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":377,"threads_started":1}
I20260924 00:52:52.684310 14625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:52:52 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:52:52.699743 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.685092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:42000)
0924 00:52:52.685325 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.685344 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:52.685456 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:52.686466 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:52.686475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:52.686907 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:52.687076 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:52.687086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.688418 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.688423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:52.689468 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:52.689480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.690762 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.690766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:52.690916 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:52.691518 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:52.691535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:52.693677 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:52.697218 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:52.697224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:52.697228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:52.697590 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:52.697975 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:52.697979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:52.697981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:52.698124 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:52.698802 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:52.698807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:52.698953 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 00:52:52.699062 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:52:52.699352 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 00:52:52.699815 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.685334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.0.222.61:39499)
0924 00:52:52.686019 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:52.686026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:52.686051 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:52:52.686096 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:52.686102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:52.686307 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:52.686419 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:52.687236 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.688284 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.689616 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.690583 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.690769 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:52.693930 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:52.693948 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:52.693968 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:52.694023 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:52.697062 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:52.697731 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:52.697738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:52.697740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:52.697874 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:52.698240 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:52.698245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:52.698248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:52.698682 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:52.698893 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:52.699248 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 00:52:52.699463 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":407,"threads_started":1}
I20260924 00:52:52.702251 14625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:52:52.703474 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.702982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42102)
0924 00:52:52.703181 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.703322 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 00:52:52.703876 14577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
I20260924 00:52:52.705530 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.705134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42112)
0924 00:52:52.705309 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.705399 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 00:52:52.705914 14625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab.
I20260924 00:52:52.713567 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:46013
--webserver_interface=127.0.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:33375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42429,127.0.222.61:39499,127.0.222.60:46013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:52:52.755264 14625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 1
I20260924 00:52:52.756558 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.756072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42122)
0924 00:52:52.756287 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.756405 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 00:52:52.757750 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.757333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42124)
0924 00:52:52.757535 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.757622 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 00:52:52.757994 14625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
I20260924 00:52:52.826212 14625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 2
I20260924 00:52:52.827353 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.826894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42136)
0924 00:52:52.827087 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.827181 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 00:52:52.828560 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.828163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42148)
0924 00:52:52.828348 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.828428 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260924 00:52:52.828792 14625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
I20260924 00:52:52.861617 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } attempt: 5
Sep 24 00:52:52 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:52:52.876842 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.862495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.0.222.61:39499)
0924 00:52:52.862649 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:52.862653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:52.862674 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:52:52.862889 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:52.862893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:52.862977 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:52.863056 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:52.864103 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.864835 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.866626 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.867092 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.867214 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:52.871154 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:52.871159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:52.871163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:52.871202 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:52.873880 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:52.874837 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:52.874842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:52.874845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:52.874980 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:52.875429 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:52.875433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:52.875434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:52.875716 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:52.875833 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:52.876383 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 00:52:52.876578 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:52:52.877012 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.862338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:42016)
0924 00:52:52.862564 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.862588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:52.862741 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:52.863134 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:52.863140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:52.863678 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:52.863922 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:52.863935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.864973 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.864979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:52.866454 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:52.866471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:52.867287 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:52.867293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:52.867492 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:52.868145 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:52.868170 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:52.870898 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:52.874069 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:52.874075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:52.874080 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:52.874682 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:52.875107 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:52.875113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:52.875117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:52.875324 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:52.875837 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:52.875843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:52.876037 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 00:52:52.876262 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 00:52:52.876627 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 00:52:52.878120 14551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:52:52.879459 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.878891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42150)
0924 00:52:52.879116 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.879227 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 00:52:52.880573 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.880189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42166)
0924 00:52:52.880338 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.880446 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 00:52:52.880849 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
I20260924 00:52:52.920028 14625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 3
I20260924 00:52:52.921293 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.920756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42168)
0924 00:52:52.921007 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.921098 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 00:52:52.922369 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.921949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42176)
0924 00:52:52.922159 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.922237 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260924 00:52:52.922564 14625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
I20260924 00:52:52.932088 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 1
I20260924 00:52:52.933324 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.932715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42180)
0924 00:52:52.932940 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.933075 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 00:52:52.934453 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:52.934063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42190)
0924 00:52:52.934222 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:52.934321 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 00:52:52.934686 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
I20260924 00:52:53.002947 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 2
I20260924 00:52:53.004212 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.003641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42194)
0924 00:52:53.003898 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.004009 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 00:52:53.005465 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.005025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42200)
0924 00:52:53.005185 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.005321 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 00:52:53.005798 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
W20260924 00:52:53.052878 14632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:53.053176 14632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:53.053332 14632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:53.060724 14632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:52:53.060844 14632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:53.060918 14632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:53.060993 14632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:52:53.061053 14632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:52:53.065748 14625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 4
I20260924 00:52:53.066900 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.066418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42206)
0924 00:52:53.066650 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.066742 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 00:52:53.067830 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.067514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42214)
0924 00:52:53.067653 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.067724 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 00:52:53.068030 14625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
I20260924 00:52:53.071586 14632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:33375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42429,127.0.222.61:39499,127.0.222.60:46013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:46013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:53.072929 14632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:53.074280 14632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:53.086100 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:53.086129 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:53.086054 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:53.086660 14632 server_base.cc:1034] running on GCE node
I20260924 00:52:53.087474 14632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:53.088946 14632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:53.090144 14632 hybrid_clock.cc:648] HybridClock initialized: now 1790211173090085 us; error 52 us; skew 500 ppm
I20260924 00:52:53.090467 14632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:53.093573 14632 init.cc:377] Logged in from keytab as kudu/127.0.222.60@KRBTEST.COM (short username kudu)
I20260924 00:52:53.095005 14632 webserver.cc:460] Webserver started at http://127.0.222.60:33603/ using document root <none> and password file <none>
I20260924 00:52:53.095652 14632 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:53.095773 14632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:53.096041 14632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:52:53.098968 14632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/data/instance:
uuid: "be365f76cd094728ac53882de2a8a8b0"
format_stamp: "Formatted at 2026-09-24 00:52:53 on dist-test-slave-rtvz"
server_key: "507a54ae7b0c5b262635988053db2e12"
server_key_iv: "e4a76e876d2a36ed528d606515ffaf8c"
server_key_version: "encryptionkey@0"
I20260924 00:52:53.099674 14632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/wal/instance:
uuid: "be365f76cd094728ac53882de2a8a8b0"
format_stamp: "Formatted at 2026-09-24 00:52:53 on dist-test-slave-rtvz"
server_key: "507a54ae7b0c5b262635988053db2e12"
server_key_iv: "e4a76e876d2a36ed528d606515ffaf8c"
server_key_version: "encryptionkey@0"
I20260924 00:52:53.104682 14632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 00:52:53.108175 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:53.109629 14632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 00:52:53.109761 14632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/wal
uuid: "be365f76cd094728ac53882de2a8a8b0"
format_stamp: "Formatted at 2026-09-24 00:52:53 on dist-test-slave-rtvz"
server_key: "507a54ae7b0c5b262635988053db2e12"
server_key_iv: "e4a76e876d2a36ed528d606515ffaf8c"
server_key_version: "encryptionkey@0"
I20260924 00:52:53.109931 14632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:53.112016 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 3
I20260924 00:52:53.113080 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.112643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42216)
0924 00:52:53.112826 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.112918 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 00:52:53.114168 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.113799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42230)
0924 00:52:53.113988 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.114061 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260924 00:52:53.114456 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:46013: Network error: Client connection negotiation failed: client connection to 127.0.222.60:46013: connect: Connection refused (error 111)
I20260924 00:52:53.123920 14632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:53.128294 14632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:53.128571 14632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:53.146715 14632 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:46013
I20260924 00:52:53.146754 14699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:46013 every 8 connection(s)
I20260924 00:52:53.148103 14632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
I20260924 00:52:53.151764 14700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:52:53.155007 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 14632
I20260924 00:52:53.155186 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790210962623352-888-0/minicluster-data/master-2/wal/instance
I20260924 00:52:53.155649 888 external_mini_cluster.cc:1428] Setting key 7a507e845126710c0c1fb2aa79f10438
I20260924 00:52:53.159775 14700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260924 00:52:53.180197 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.164087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.0.222.62:42429)
0924 00:52:53.164279 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.164285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.165406 (+ 1121us) server_negotiation.cc:407] Connection header received
0924 00:52:53.166436 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.166441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.166527 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.166635 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.167920 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.168615 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.170099 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.170516 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.170623 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.174423 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.174431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.174435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.174470 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.177246 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.177978 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.177982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.177985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.178108 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.178520 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.178523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.178525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.178864 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.179015 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.179778 (+ 763us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.179943 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:52:53.181517 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.157751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.0.222.60:46013)
0924 00:52:53.163722 (+ 5971us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.163733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.163770 (+ 37us) server_negotiation.cc:407] Connection header received
0924 00:52:53.166140 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.166169 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.166463 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.166640 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.168213 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.169420 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.170721 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.171472 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.171595 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.175829 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.175837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.175850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.175887 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.178489 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.179202 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.179209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.179212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.179360 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.179668 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.179672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.179675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.180108 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.180375 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.180723 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.180957 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":221,"server-negotiator.queue_time_us":5726,"thread_start_us":5512,"threads_started":1}
I20260924 00:52:53.181551 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.164312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42429 (local address 127.0.0.1:49196)
0924 00:52:53.165256 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.165312 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:53.166155 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:53.166780 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:53.166786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:53.167369 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:53.167759 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.167777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.168769 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.168772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.169973 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.169984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.170730 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.170737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.171000 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.171633 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:53.171660 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:53.174181 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:53.177367 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.177378 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.177390 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.177861 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.178208 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.178213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.178217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.178411 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.178985 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:53.178991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:53.179397 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 00:52:53.179657 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 00:52:53.180162 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":26,"thread_start_us":372,"threads_started":1}
I20260924 00:52:53.183938 14700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:53.196543 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.184665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.0.222.61:39499)
0924 00:52:53.184872 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.184886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.185062 (+ 176us) server_negotiation.cc:407] Connection header received
0924 00:52:53.185224 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.185230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.185297 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.185373 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.186057 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.186720 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.187819 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.188245 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.188357 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.191382 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.191387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.191391 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.191424 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.194139 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.194765 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.194770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.194772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.194927 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.195265 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.195270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.195273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.195531 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.195670 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.196060 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.196220 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 00:52:53.196576 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.184634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:39499 (local address 127.0.0.1:42030)
0924 00:52:53.184980 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.184992 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:53.185092 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:53.185414 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:53.185418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:53.185782 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:53.185922 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.185930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.186841 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.186846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.187696 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.187706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.188377 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.188383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.188572 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.189063 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:53.189079 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:53.191205 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:53.194264 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.194270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.194274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.194658 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.195029 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.195034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.195038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.195169 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.195639 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:53.195643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:53.195800 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 00:52:53.195964 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 00:52:53.196239 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268}
I20260924 00:52:53.197855 14700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:52:53.198563 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.184668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.0.222.62:42429)
0924 00:52:53.184816 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.184821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.184837 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:52:53.184993 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.184999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.185060 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.185135 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.186331 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.187055 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.188285 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.188680 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.188780 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.193008 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.193017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.193021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.193061 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.195653 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.196466 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.196471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.196474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.196628 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.197093 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.197098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.197102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.197494 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.197751 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.198087 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.198267 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:52:53.211925 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.198726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42244)
0924 00:52:53.198868 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.198885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:53.198976 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:53.199483 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:53.199492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:53.200046 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:53.200202 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.200214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.201299 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.201304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.202386 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.202397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.203250 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.203256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.203431 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.203992 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:53.204013 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:53.206372 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:53.209638 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.209643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.209647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.210020 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.210418 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.210421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.210423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.210543 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.211135 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:53.211138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:53.211265 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 00:52:53.211397 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 00:52:53.211634 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 00:52:53.212167 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.199067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.0.222.60:46013)
0924 00:52:53.199203 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.199209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.199225 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:52:53.199264 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.199268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.199330 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.199425 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.200350 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.201161 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.202515 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.203119 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.203213 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.206591 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.206596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.206599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.206630 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.209531 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.210133 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.210139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.210142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.210306 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.210642 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.210647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.210651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.211016 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.211156 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.211509 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.211810 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":37}
I20260924 00:52:53.215693 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.200250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.0.222.61:39499)
0924 00:52:53.200413 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.200418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.200435 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:52:53.200518 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.200523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.200590 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.200681 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.201952 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.202800 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.204335 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.204748 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.204840 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.209446 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.209451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.209454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.209488 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.212852 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.213711 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.213715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.213717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.213871 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.214272 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.214278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.214281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.214598 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.214895 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.215163 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.215379 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:52:53.217603 14700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: Bootstrap starting.
I20260924 00:52:53.220631 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:52:53.221505 14700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:53.222543 14700 log.cc:824] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:53.225410 14700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: No bootstrap required, opened a new log
I20260924 00:52:53.235471 14700 raft_consensus.cc:348] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.236162 14700 raft_consensus.cc:374] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:53.236351 14700 raft_consensus.cc:729] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be365f76cd094728ac53882de2a8a8b0, State: Initialized, Role: FOLLOWER
I20260924 00:52:53.237323 14700 consensus_queue.cc:260] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.242564 14700 sys_catalog.cc:564] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:52:53.245666 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.225342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.0.222.62:42429)
0924 00:52:53.225527 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.225532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.227197 (+ 1665us) server_negotiation.cc:407] Connection header received
0924 00:52:53.227468 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.227474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.227560 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.227666 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.230768 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.231865 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.234130 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.234812 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.234913 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.239071 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.239080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.239085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.239144 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.242467 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.243288 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.243294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.243298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.243463 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.243883 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.243895 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.243898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.244268 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.244495 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.245039 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.245323 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:52:53.246706 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.229516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.0.222.61:39499)
0924 00:52:53.229693 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.229698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.229717 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:52:53.229774 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.229778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.229839 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.229928 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.231995 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.233230 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.235412 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.236032 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.236156 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.239799 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.239806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.239811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.239857 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.242974 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.243783 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.243789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.243792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.243963 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.244384 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.244389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.244392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.244774 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.244999 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.246116 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.246365 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:52:53.248180 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.226213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.0.222.60:46013)
0924 00:52:53.226374 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.226379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.229933 (+ 3554us) server_negotiation.cc:407] Connection header received
0924 00:52:53.230112 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.230118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.230194 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.230265 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.232434 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.233624 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.235705 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.236367 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.236463 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.240728 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.240734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.240738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.240777 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.244575 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.245535 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.245542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.245546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.245759 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.246222 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.246228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.246231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.246686 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.246943 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.247501 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.247749 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:52:53.248653 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.249013 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:52:53.258745 14724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:52:53.258987 14724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:52:53.255141 14632 master_runner.cc:186] Error getting master registration for 127.0.222.62:42429: OK, instance_id { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" instance_seqno: 1790211172151238 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:52:53.264384 14632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:52:53.291656 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 4
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260924 00:52:53.306787 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.292287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.0.222.60:46013)
0924 00:52:53.292454 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.292463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.292484 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:52:53.292664 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.292732 (+ 68us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.292820 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.292902 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.293755 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.294537 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.295758 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.296222 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.296330 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.300229 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.300234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.300237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.300284 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.304196 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.304815 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.304819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.304822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.304978 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.305365 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.305373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.305375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.305662 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.305796 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.306189 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.306414 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:52:53.306807 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.292222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42254)
0924 00:52:53.292415 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.292435 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:53.292547 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:53.292956 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:53.292960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:53.293437 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:53.293608 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.293617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.294662 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.294668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.295621 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.295633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.296355 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.296360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.296543 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.297266 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:53.297289 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:53.300019 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:53.304323 (+ 4304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.304329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.304332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.304700 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.305085 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.305089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.305092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.305262 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.305774 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:53.305778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:53.305942 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 00:52:53.306054 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 00:52:53.306414 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 00:52:53.311417 14551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2: Bootstrap starting.
I20260924 00:52:53.314714 14551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:53.316033 14551 log.cc:824] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:53.319456 14551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2: No bootstrap required, opened a new log
I20260924 00:52:53.326685 14551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.327018 14551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:53.327136 14551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a209e67f9744eb8d2746fe481de2c2, State: Initialized, Role: FOLLOWER
I20260924 00:52:53.327832 14551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.329596 14727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.329995 14727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:52:53.330111 14551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 00:52:53.339869 14738 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:52:53.340096 14738 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:52:53.339660 14483 master_runner.cc:186] Error getting master registration for 127.0.222.61:39499: OK, instance_id { permanent_uuid: "691fb3656720438b95b649bd05b30552" instance_seqno: 1790211172642452 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:52:53.353820 14483 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:52:53.367241 14625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } attempt: 5
I20260924 00:52:53.367771 14727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:52:53.368111 14727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.369897 14727 leader_election.cc:288] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 691fb3656720438b95b649bd05b30552 (127.0.222.61:39499), be365f76cd094728ac53882de2a8a8b0 (127.0.222.60:46013)
I20260924 00:52:53.371389 14675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67a209e67f9744eb8d2746fe481de2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be365f76cd094728ac53882de2a8a8b0" is_pre_election: true
I20260924 00:52:53.372066 14675 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a209e67f9744eb8d2746fe481de2c2 in term 0.
I20260924 00:52:53.372789 14503 leader_election.cc:302] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a209e67f9744eb8d2746fe481de2c2, be365f76cd094728ac53882de2a8a8b0; no voters:
I20260924 00:52:53.373270 14727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:52:53.373456 14727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:52:53.373560 14727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:52:53.376865 14727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.378095 14675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67a209e67f9744eb8d2746fe481de2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be365f76cd094728ac53882de2a8a8b0"
I20260924 00:52:53.378320 14675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:52:53.380894 14654 tablet.cc:2369] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:52:53.383284 14675 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a209e67f9744eb8d2746fe481de2c2 in term 1.
I20260924 00:52:53.383646 14727 leader_election.cc:288] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [CANDIDATE]: Term 1 election: Requested vote from peers 691fb3656720438b95b649bd05b30552 (127.0.222.61:39499), be365f76cd094728ac53882de2a8a8b0 (127.0.222.60:46013)
Sep 24 00:52:53 dist-test-slave-rtvz krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260924 00:52:53.384143 14503 leader_election.cc:302] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a209e67f9744eb8d2746fe481de2c2, be365f76cd094728ac53882de2a8a8b0; no voters:
I20260924 00:52:53.385937 14727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 00:52:53.391498 14505 tablet.cc:2369] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:52:53.396201 14727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 67a209e67f9744eb8d2746fe481de2c2, State: Running, Role: LEADER
I20260924 00:52:53.396849 14727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.399567 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.368620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.0.222.60:46013)
0924 00:52:53.368803 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:53.368809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:53.368827 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:52:53.368893 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:53.368899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:53.368973 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:53.369066 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:53.371275 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.372534 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.374657 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.375360 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.375472 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.386291 (+ 10819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:53.386298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:53.386304 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:53.386347 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:53.391763 (+ 5416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.393448 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.393457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.393461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.393739 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:53.394504 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:53.394509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:53.394546 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:53.395370 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:53.396032 (+ 662us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:53.396818 (+ 786us) server_negotiation.cc:299] Negotiation successful
0924 00:52:53.397998 (+ 1180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:52:53.399601 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:53.367892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:46013 (local address 127.0.0.1:42262)
0924 00:52:53.368249 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:53.368270 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:53.368390 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:53.369421 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:53.369426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:53.369973 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:53.370191 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.370203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.372864 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.372871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.374313 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:53.374329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:53.375706 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:53.375713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:53.375944 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:53.376714 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:53.376740 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:53.385622 (+ 8882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:53.392134 (+ 6512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.392140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.392144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.393149 (+ 1005us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.394020 (+ 871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:53.394025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:53.394029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:53.394329 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:53.395560 (+ 1231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:53.395566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:53.396055 (+ 489us) client_negotiation.cc:769] Sending connection context
0924 00:52:53.396769 (+ 714us) client_negotiation.cc:240] Negotiation successful
0924 00:52:53.398201 (+ 1432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260924 00:52:53.410104 14625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552: Bootstrap starting.
I20260924 00:52:53.410034 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a209e67f9744eb8d2746fe481de2c2. Latest consensus state: current_term: 1 leader_uuid: "67a209e67f9744eb8d2746fe481de2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.411015 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:53.416038 14625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:53.417551 14625 log.cc:824] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:53.422653 14742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:52:53.425278 14625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552: No bootstrap required, opened a new log
I20260924 00:52:53.435395 14742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:52:53.445961 14625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.446893 14625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:53.447027 14625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691fb3656720438b95b649bd05b30552, State: Initialized, Role: FOLLOWER
I20260924 00:52:53.447728 14625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } }
I20260924 00:52:53.454051 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.454447 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:52:53.456017 14625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:52:53.456064 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67a209e67f9744eb8d2746fe481de2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fb3656720438b95b649bd05b30552"
I20260924 00:52:53.457238 14599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:52:53.456207 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67a209e67f9744eb8d2746fe481de2c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691fb3656720438b95b649bd05b30552" is_pre_election: true
I20260924 00:52:53.463347 14675 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [term 1 FOLLOWER]: Refusing update from remote peer 67a209e67f9744eb8d2746fe481de2c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:52:53.464862 14727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:52:53.471800 14599 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a209e67f9744eb8d2746fe481de2c2 in term 1.
I20260924 00:52:53.472680 14600 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [term 1 FOLLOWER]: Refusing update from remote peer 67a209e67f9744eb8d2746fe481de2c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:52:53.477828 14727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:52:53.494297 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a209e67f9744eb8d2746fe481de2c2. Latest consensus state: current_term: 1 leader_uuid: "67a209e67f9744eb8d2746fe481de2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.494621 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:52:53.499138 14727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67a209e67f9744eb8d2746fe481de2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.499481 14727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:53.516299 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a209e67f9744eb8d2746fe481de2c2. Latest consensus state: current_term: 1 leader_uuid: "67a209e67f9744eb8d2746fe481de2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.516636 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:52:53.517565 14742 catalog_manager.cc:1347] Generated new cluster ID: 61cdc68f89084efebcba438ceb31edbe
I20260924 00:52:53.517683 14742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:52:53.521638 14762 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 00:52:53.536582 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67a209e67f9744eb8d2746fe481de2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.536899 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:53.541671 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67a209e67f9744eb8d2746fe481de2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.541970 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:52:53.544749 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67a209e67f9744eb8d2746fe481de2c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a209e67f9744eb8d2746fe481de2c2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42429 } } peers { permanent_uuid: "691fb3656720438b95b649bd05b30552" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 39499 } } peers { permanent_uuid: "be365f76cd094728ac53882de2a8a8b0" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 46013 } } }
I20260924 00:52:53.545048 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:52:53.549986 14742 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:52:53.551872 14742 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:52:53.572082 14742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a209e67f9744eb8d2746fe481de2c2: Generated new TSK 0
I20260924 00:52:53.573341 14742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:52:54.261067 14724 catalog_manager.cc:1259] Loaded cluster ID: 61cdc68f89084efebcba438ceb31edbe
I20260924 00:52:54.261174 14724 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: loading cluster ID for follower catalog manager: success
I20260924 00:52:54.263590 14724 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: acquiring CA information for follower catalog manager: success
I20260924 00:52:54.265393 14724 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be365f76cd094728ac53882de2a8a8b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:52:54.526026 14762 catalog_manager.cc:1259] Loaded cluster ID: 61cdc68f89084efebcba438ceb31edbe
I20260924 00:52:54.526144 14762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552: loading cluster ID for follower catalog manager: success
I20260924 00:52:54.528743 14762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552: acquiring CA information for follower catalog manager: success
I20260924 00:52:54.531054 14762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 691fb3656720438b95b649bd05b30552: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:52:54.667258 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 14483
I20260924 00:52:54.668928 14546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:54.808735 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 14483
I20260924 00:52:54.829130 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 14558
I20260924 00:52:54.830461 14620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:54.946213 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 14558
I20260924 00:52:54.961814 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 14632
I20260924 00:52:54.962893 14695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:55.073515 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 14632
2026-09-24T00:52:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4613 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:52:55 dist-test-slave-rtvz krb5kdc[14776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:52:55 dist-test-slave-rtvz krb5kdc[14776](info): set up 2 sockets
Sep 24 00:52:55 dist-test-slave-rtvz krb5kdc[14776](info): commencing operation
krb5kdc: starting...
W20260924 00:52:56.096894 888 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:52:56 dist-test-slave-rtvz krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:52:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:52:56.326701 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33891
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:52:56.654865 14789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:56.655190 14789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:56.655350 14789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:56.662810 14789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:52:56.662941 14789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:56.663024 14789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:56.663123 14789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:52:56.663187 14789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:52:56.673815 14789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:56.675230 14789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:56.676791 14789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:56.689898 14795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:56.689877 14798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:56.689873 14794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:56.690791 14789 server_base.cc:1034] running on GCE node
I20260924 00:52:56.691602 14789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:56.693094 14789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:56.694337 14789 hybrid_clock.cc:648] HybridClock initialized: now 1790211176694270 us; error 46 us; skew 500 ppm
I20260924 00:52:56.694824 14789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:56 dist-test-slave-rtvz krb5kdc[14776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:56.698137 14789 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:52:56.699759 14789 webserver.cc:460] Webserver started at http://127.0.222.62:36011/ using document root <none> and password file <none>
I20260924 00:52:56.700491 14789 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:56.700610 14789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:56.700910 14789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:52:56.704047 14789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe"
format_stamp: "Formatted at 2026-09-24 00:52:56 on dist-test-slave-rtvz"
server_key: "3d9e38d9017762dcf910ce59342eadf5"
server_key_iv: "4cc86a59b81b92148f87ce086009f1f8"
server_key_version: "encryptionkey@0"
I20260924 00:52:56.704849 14789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe"
format_stamp: "Formatted at 2026-09-24 00:52:56 on dist-test-slave-rtvz"
server_key: "3d9e38d9017762dcf910ce59342eadf5"
server_key_iv: "4cc86a59b81b92148f87ce086009f1f8"
server_key_version: "encryptionkey@0"
I20260924 00:52:56.710516 14789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 00:52:56.714433 14805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:56.716080 14789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:52:56.716321 14789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe"
format_stamp: "Formatted at 2026-09-24 00:52:56 on dist-test-slave-rtvz"
server_key: "3d9e38d9017762dcf910ce59342eadf5"
server_key_iv: "4cc86a59b81b92148f87ce086009f1f8"
server_key_version: "encryptionkey@0"
I20260924 00:52:56.716536 14789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:56.743378 14789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:56.748083 14789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:56.748420 14789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:56.766269 14789 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33891
I20260924 00:52:56.766270 14856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33891 every 8 connection(s)
I20260924 00:52:56.767759 14789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:52:56.768901 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 14789
I20260924 00:52:56.769109 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:52:56.769810 888 external_mini_cluster.cc:1428] Setting key 17b412f32b5d48f6d33ae4731e0487df
I20260924 00:52:56.772740 14857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:52:56.783459 14857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe: Bootstrap starting.
Sep 24 00:52:56 dist-test-slave-rtvz krb5kdc[14776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:56.786877 14857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:56.787976 14857 log.cc:824] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:56.790900 14857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe: No bootstrap required, opened a new log
I20260924 00:52:56.794480 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:56.771745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.222.62:33891)
0924 00:52:56.772600 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:56.772619 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:56.773039 (+ 420us) server_negotiation.cc:407] Connection header received
0924 00:52:56.775029 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:56.775053 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:56.775516 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:56.775937 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:56.777680 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:56.779052 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:56.780626 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:56.781488 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:56.781677 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:56.786268 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:56.786278 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:56.786294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:56.786349 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:56.789672 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:56.790557 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:56.790565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:56.790572 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:56.790767 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:56.791274 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:56.791279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:56.791281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:56.791906 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:56.792204 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:56.792777 (+ 573us) server_negotiation.cc:299] Negotiation successful
0924 00:52:56.793317 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":356,"threads_started":1}
I20260924 00:52:56.798746 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:52:56.799070 14857 raft_consensus.cc:348] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33891 } }
I20260924 00:52:56.799439 14857 raft_consensus.cc:374] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:56.799526 14857 raft_consensus.cc:729] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33fd9edb7dc49b1839ee8a5e4dadebe, State: Initialized, Role: FOLLOWER
I20260924 00:52:56.800125 14857 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33891 } }
I20260924 00:52:56.800343 14857 raft_consensus.cc:388] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:52:56.800467 14857 raft_consensus.cc:482] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:52:56.800621 14857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:52:56.803174 14857 raft_consensus.cc:504] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33891 } }
I20260924 00:52:56.803692 14857 leader_election.cc:302] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d33fd9edb7dc49b1839ee8a5e4dadebe; no voters:
I20260924 00:52:56.804298 14857 leader_election.cc:288] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:52:56.804431 14862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:52:56.805187 14862 raft_consensus.cc:686] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [term 1 LEADER]: Becoming Leader. State: Replica: d33fd9edb7dc49b1839ee8a5e4dadebe, State: Running, Role: LEADER
I20260924 00:52:56.805749 14862 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33891 } }
I20260924 00:52:56.806321 14857 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:52:56.808548 14863 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33891 } } }
I20260924 00:52:56.809036 14863 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:56.809489 14864 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33fd9edb7dc49b1839ee8a5e4dadebe. Latest consensus state: current_term: 1 leader_uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33fd9edb7dc49b1839ee8a5e4dadebe" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33891 } } }
I20260924 00:52:56.809765 14864 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:56.811380 14872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:52:56.815624 14872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:52:56.825086 14872 catalog_manager.cc:1347] Generated new cluster ID: df267fd9f4f0409ab4cadfb19e5101cb
I20260924 00:52:56.825240 14872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:52:56.841912 14872 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:52:56.843852 14872 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:52:56.851131 14872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d33fd9edb7dc49b1839ee8a5e4dadebe: Generated new TSK 0
I20260924 00:52:56.851985 14872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:52:56.859952 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 14789
I20260924 00:52:56.861307 14852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:52:56.951532 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 14789
2026-09-24T00:52:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1874 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:52:57 dist-test-slave-rtvz krb5kdc[14886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:52:57 dist-test-slave-rtvz krb5kdc[14886](info): set up 2 sockets
Sep 24 00:52:57 dist-test-slave-rtvz krb5kdc[14886](info): commencing operation
krb5kdc: starting...
W20260924 00:52:57.964839 888 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:52:58 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:52:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:52:58.194221 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42639
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:52:58.515522 14899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:58.515836 14899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:58.515975 14899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:58.523263 14899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:52:58.523373 14899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:58.523442 14899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:58.523521 14899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:52:58.523583 14899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:52:58.533490 14899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:58.534816 14899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:58.536100 14899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:58.547727 14905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:58.547905 14904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:58.547868 14907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:58.549130 14899 server_base.cc:1034] running on GCE node
I20260924 00:52:58.550026 14899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:58.551574 14899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:58.552824 14899 hybrid_clock.cc:648] HybridClock initialized: now 1790211178552742 us; error 72 us; skew 500 ppm
I20260924 00:52:58.553172 14899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:58 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:58.559834 14899 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:52:58.561517 14899 webserver.cc:460] Webserver started at http://127.0.222.62:33927/ using document root <none> and password file <none>
I20260924 00:52:58.562256 14899 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:58.562388 14899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:58.562680 14899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:52:58.565940 14899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "04ce98d104fb4061b530edbae7654c03"
format_stamp: "Formatted at 2026-09-24 00:52:58 on dist-test-slave-rtvz"
server_key: "283d35ab61044792c2498271c6283de7"
server_key_iv: "bb7517ea2a82cd6239379bf791b161c2"
server_key_version: "encryptionkey@0"
I20260924 00:52:58.566777 14899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "04ce98d104fb4061b530edbae7654c03"
format_stamp: "Formatted at 2026-09-24 00:52:58 on dist-test-slave-rtvz"
server_key: "283d35ab61044792c2498271c6283de7"
server_key_iv: "bb7517ea2a82cd6239379bf791b161c2"
server_key_version: "encryptionkey@0"
I20260924 00:52:58.572403 14899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 00:52:58.576296 14914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:58.578063 14899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:52:58.578258 14899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "04ce98d104fb4061b530edbae7654c03"
format_stamp: "Formatted at 2026-09-24 00:52:58 on dist-test-slave-rtvz"
server_key: "283d35ab61044792c2498271c6283de7"
server_key_iv: "bb7517ea2a82cd6239379bf791b161c2"
server_key_version: "encryptionkey@0"
I20260924 00:52:58.578465 14899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:58.591840 14899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:58.596403 14899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:58.596688 14899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:58.614766 14899 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42639
I20260924 00:52:58.614806 14966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42639 every 8 connection(s)
I20260924 00:52:58.616154 14899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:52:58.616569 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 14899
I20260924 00:52:58.616767 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:52:58.617739 888 external_mini_cluster.cc:1428] Setting key 02171f814b2e6db8e863a85bec0217cd
I20260924 00:52:58.620884 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 00:52:58 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:58.634197 14967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Bootstrap starting.
I20260924 00:52:58.638442 14967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Neither blocks nor log segments found. Creating new log.
I20260924 00:52:58.639673 14967 log.cc:824] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Log is configured to *not* fsync() on all Append() calls
I20260924 00:52:58.642890 14970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:58.619885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.0.222.62:42639)
0924 00:52:58.620695 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:58.620708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:58.620753 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:52:58.622462 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:58.622486 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:58.622891 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:58.623275 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:58.625068 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:58.626608 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:58.628435 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:58.629162 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:58.629379 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:58.634183 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:58.634196 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:58.634216 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:58.634266 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:58.638000 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:58.638896 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:58.638905 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:58.638914 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:58.639125 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:58.639507 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:58.639512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:58.639515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:58.640252 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:58.640533 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:58.641085 (+ 552us) server_negotiation.cc:299] Negotiation successful
0924 00:52:58.641569 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":398,"threads_started":1}
I20260924 00:52:58.643357 14967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: No bootstrap required, opened a new log
I20260924 00:52:58.651448 14967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:52:58.651822 14967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:52:58.651904 14967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ce98d104fb4061b530edbae7654c03, State: Initialized, Role: FOLLOWER
I20260924 00:52:58.652458 14967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:52:58.652678 14967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:52:58.652772 14967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:52:58.652900 14967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:52:58.655088 14967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:52:58.655650 14967 leader_election.cc:302] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04ce98d104fb4061b530edbae7654c03; no voters:
I20260924 00:52:58.656210 14967 leader_election.cc:288] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:52:58.656330 14972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:52:58.656924 14972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 1 LEADER]: Becoming Leader. State: Replica: 04ce98d104fb4061b530edbae7654c03, State: Running, Role: LEADER
I20260924 00:52:58.657516 14972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:52:58.658223 14967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:52:58.660053 14973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04ce98d104fb4061b530edbae7654c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } } }
I20260924 00:52:58.660501 14973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:58.660346 14974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04ce98d104fb4061b530edbae7654c03. Latest consensus state: current_term: 1 leader_uuid: "04ce98d104fb4061b530edbae7654c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } } }
I20260924 00:52:58.660605 14974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: This master's current role is: LEADER
I20260924 00:52:58.661437 14979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:52:58.667800 14979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:52:58.677016 14979 catalog_manager.cc:1347] Generated new cluster ID: 4f797f09e0094f91a4e7dc4864cc0467
I20260924 00:52:58.677148 14979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:52:58.693984 14979 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:52:58.695354 14979 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:52:58.706104 14979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Generated new TSK 0
I20260924 00:52:58.706979 14979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:52:58.772312 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42639
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:52:59.109122 14993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:59.109535 14993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:59.109658 14993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:59.117107 14993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:59.117287 14993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:59.117458 14993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:52:59.127585 14993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:59.129047 14993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:59.130474 14993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:59.143797 14999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:59.143834 15001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:59.143774 14998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:59.144569 14993 server_base.cc:1034] running on GCE node
I20260924 00:52:59.145283 14993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:59.146260 14993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:59.147455 14993 hybrid_clock.cc:648] HybridClock initialized: now 1790211179147402 us; error 47 us; skew 500 ppm
I20260924 00:52:59.147753 14993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:59 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:59.150867 14993 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:52:59.152299 14993 webserver.cc:460] Webserver started at http://127.0.222.1:37897/ using document root <none> and password file <none>
I20260924 00:52:59.152958 14993 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:59.153059 14993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:59.153362 14993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:52:59.156317 14993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "c6ca92cd60d9414cbf8c933e8de1065a"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "4716b35ec88ff7a67d072d7922c81ec7"
server_key_iv: "505c042b520c70ba649313711594cef0"
server_key_version: "encryptionkey@0"
I20260924 00:52:59.157063 14993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "c6ca92cd60d9414cbf8c933e8de1065a"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "4716b35ec88ff7a67d072d7922c81ec7"
server_key_iv: "505c042b520c70ba649313711594cef0"
server_key_version: "encryptionkey@0"
I20260924 00:52:59.162380 14993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 00:52:59.165915 15008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:59.167393 14993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:52:59.167547 14993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "c6ca92cd60d9414cbf8c933e8de1065a"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "4716b35ec88ff7a67d072d7922c81ec7"
server_key_iv: "505c042b520c70ba649313711594cef0"
server_key_version: "encryptionkey@0"
I20260924 00:52:59.167732 14993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:59.182441 14993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:59.186743 14993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:59.187028 14993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:59.187965 14993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:52:59.189342 14993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:52:59.189427 14993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:59.189550 14993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:52:59.189608 14993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:59.220338 14993 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:35071
I20260924 00:52:59.220405 15121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:35071 every 8 connection(s)
I20260924 00:52:59.221653 14993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:52:59.226337 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 14993
I20260924 00:52:59.226521 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:52:59.227164 888 external_mini_cluster.cc:1428] Setting key 6d3c9974e2a5dd8c572d075308e234ed
Sep 24 00:52:59 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:59.242626 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:59.224158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:33975 (local address 127.0.222.62:42639)
0924 00:52:59.224624 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:59.224630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:59.225667 (+ 1037us) server_negotiation.cc:407] Connection header received
0924 00:52:59.227527 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:59.227533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:59.227619 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:59.227748 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:59.229595 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:59.230593 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:59.232254 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:59.232802 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:59.232933 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:59.236547 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:59.236553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:59.236556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:59.236592 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:59.239359 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:59.240245 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:59.240250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:59.240252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:59.240399 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:59.240848 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:59.240851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:59.240853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:59.241138 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:59.241294 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:59.242159 (+ 865us) server_negotiation.cc:299] Negotiation successful
0924 00:52:59.242365 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":220,"threads_started":1}
I20260924 00:52:59.244028 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:59.224754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:33975)
0924 00:52:59.225505 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:59.225545 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:59.227228 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:59.228200 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:59.228209 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:59.228736 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:59.229404 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:59.229429 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:59.230764 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:59.230768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:59.232106 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:59.232119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:59.233009 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:59.233013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:59.233256 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:59.233935 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:59.233960 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:59.236301 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:59.239516 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:59.239525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:59.239544 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:59.240091 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:59.240495 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:59.240500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:59.240503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:59.240741 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:59.241268 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:59.241276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:59.241714 (+ 438us) client_negotiation.cc:769] Sending connection context
0924 00:52:59.242006 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 00:52:59.242542 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":265,"threads_started":1}
I20260924 00:52:59.245923 15122 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42639
I20260924 00:52:59.246268 15122 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:59.247025 15122 heartbeater.cc:511] Master 127.0.222.62:42639 requested a full tablet report, sending...
I20260924 00:52:59.249166 14931 ts_manager.cc:194] Registered new tserver with Master: c6ca92cd60d9414cbf8c933e8de1065a (127.0.222.1:35071)
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:52:59.250699 14931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:33975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:52:59.281684 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42639
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:52:59.610530 15128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:52:59.610870 15128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:52:59.611016 15128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:52:59.618986 15128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:52:59.619138 15128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:52:59.619305 15128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:52:59.629962 15128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:52:59.631290 15128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:52:59.632609 15128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:52:59.646415 15134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:59.646591 15136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:52:59.646566 15133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:52:59.647497 15128 server_base.cc:1034] running on GCE node
I20260924 00:52:59.648111 15128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:52:59.649121 15128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:52:59.650381 15128 hybrid_clock.cc:648] HybridClock initialized: now 1790211179650319 us; error 63 us; skew 500 ppm
I20260924 00:52:59.650696 15128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:52:59 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:52:59.653914 15128 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:52:59.655387 15128 webserver.cc:460] Webserver started at http://127.0.222.2:33975/ using document root <none> and password file <none>
I20260924 00:52:59.656052 15128 fs_manager.cc:359] Metadata directory not provided
I20260924 00:52:59.656167 15128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:52:59.656457 15128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:52:59.659540 15128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "7870ec61274e42e9b3c685d708fe9045"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "598817166d4143008ebe1b11aeaf3ae5"
server_key_iv: "1f1d589ca979439a3bb898555763b54c"
server_key_version: "encryptionkey@0"
I20260924 00:52:59.660323 15128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "7870ec61274e42e9b3c685d708fe9045"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "598817166d4143008ebe1b11aeaf3ae5"
server_key_iv: "1f1d589ca979439a3bb898555763b54c"
server_key_version: "encryptionkey@0"
I20260924 00:52:59.665486 15128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:52:59.669057 15143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:59.670506 15128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:52:59.670661 15128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "7870ec61274e42e9b3c685d708fe9045"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "598817166d4143008ebe1b11aeaf3ae5"
server_key_iv: "1f1d589ca979439a3bb898555763b54c"
server_key_version: "encryptionkey@0"
I20260924 00:52:59.670853 15128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:52:59.701763 15128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:52:59.706269 15128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:52:59.706599 15128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:52:59.707623 15128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:52:59.709028 15128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:52:59.709100 15128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:52:59.709184 15128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:52:59.709270 15128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 00:52:59.740564 15128 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:36281
I20260924 00:52:59.740628 15256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:36281 every 8 connection(s)
I20260924 00:52:59.742026 15128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:52:59.744377 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 15128
I20260924 00:52:59.744580 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:52:59.745244 888 external_mini_cluster.cc:1428] Setting key 73a23d3c476b692aa494313b848510cf
Sep 24 00:52:59 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:52:59.766902 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:59.744973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:42087 (local address 127.0.222.62:42639)
0924 00:52:59.746308 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0924 00:52:59.746315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:52:59.747252 (+ 937us) server_negotiation.cc:407] Connection header received
0924 00:52:59.749907 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:52:59.749912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:52:59.749984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:52:59.750078 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:52:59.752316 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:59.753044 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:59.755455 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:52:59.755921 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:52:59.756027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:59.760193 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:52:59.760201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:52:59.760206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:52:59.760251 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:52:59.763584 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:59.764459 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:59.764465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:59.764469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:59.764628 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:52:59.765056 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:52:59.765061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:52:59.765064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:52:59.765440 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:52:59.765579 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 00:52:59.766277 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 00:52:59.766511 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":1083,"threads_started":1}
I20260924 00:52:59.767783 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:52:59.745653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:42087)
0924 00:52:59.747045 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0924 00:52:59.747106 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 00:52:59.749568 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:52:59.750397 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:52:59.750410 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:52:59.751174 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:52:59.752049 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:59.752079 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:59.753254 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:59.753260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:59.755258 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:52:59.755278 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:52:59.756197 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:52:59.756206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:52:59.756556 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:52:59.757362 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:52:59.757389 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:52:59.759895 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:52:59.763774 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:59.763785 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:59.763802 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:59.764316 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:59.764722 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:52:59.764725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:52:59.764727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:52:59.764944 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:52:59.765551 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:52:59.765558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:52:59.765912 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 00:52:59.766156 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 00:52:59.766596 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":888,"threads_started":1}
I20260924 00:52:59.769287 15257 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42639
I20260924 00:52:59.769579 15257 heartbeater.cc:464] Registering TS with master...
I20260924 00:52:59.770303 15257 heartbeater.cc:511] Master 127.0.222.62:42639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
I20260924 00:52:59.772018 14931 ts_manager.cc:194] Registered new tserver with Master: 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281)
I20260924 00:52:59.772946 14931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:42087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:52:59.801343 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42639
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:00.132728 15263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:00.133045 15263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:00.133188 15263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:00.140570 15263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:00.140686 15263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:00.140810 15263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:53:00.151065 15263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:00.152437 15263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:00.153813 15263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:00.167023 15268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:00.167090 15269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:00.167063 15271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:00.168303 15263 server_base.cc:1034] running on GCE node
I20260924 00:53:00.169055 15263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:00.170122 15263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:00.171367 15263 hybrid_clock.cc:648] HybridClock initialized: now 1790211180171263 us; error 54 us; skew 500 ppm
I20260924 00:53:00.171747 15263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:00 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:00.175145 15263 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:53:00.176687 15263 webserver.cc:460] Webserver started at http://127.0.222.3:43049/ using document root <none> and password file <none>
I20260924 00:53:00.177448 15263 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:00.177570 15263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:00.177865 15263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:53:00.180927 15263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "87844157e1374003816059043822dc21"
format_stamp: "Formatted at 2026-09-24 00:53:00 on dist-test-slave-rtvz"
server_key: "d60341c58f15ab4834f786ad6cee5142"
server_key_iv: "68eb2c54385b1aec6e5cce86b2fcae6d"
server_key_version: "encryptionkey@0"
I20260924 00:53:00.181769 15263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "87844157e1374003816059043822dc21"
format_stamp: "Formatted at 2026-09-24 00:53:00 on dist-test-slave-rtvz"
server_key: "d60341c58f15ab4834f786ad6cee5142"
server_key_iv: "68eb2c54385b1aec6e5cce86b2fcae6d"
server_key_version: "encryptionkey@0"
I20260924 00:53:00.187281 15263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:53:00.191042 15278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:00.192521 15263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 00:53:00.192720 15263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "87844157e1374003816059043822dc21"
format_stamp: "Formatted at 2026-09-24 00:53:00 on dist-test-slave-rtvz"
server_key: "d60341c58f15ab4834f786ad6cee5142"
server_key_iv: "68eb2c54385b1aec6e5cce86b2fcae6d"
server_key_version: "encryptionkey@0"
I20260924 00:53:00.192945 15263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:00.207091 15263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:00.211606 15263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:00.211908 15263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:00.213001 15263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:00.214448 15263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:00.214525 15263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:00.214646 15263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:00.214704 15263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:00.245344 15263 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:36981
I20260924 00:53:00.245393 15391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:36981 every 8 connection(s)
I20260924 00:53:00.246646 15263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:53:00.253989 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 15263
I20260924 00:53:00.253983 15122 heartbeater.cc:503] Master 127.0.222.62:42639 was elected leader, sending a full tablet report...
I20260924 00:53:00.254170 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:53:00.254686 888 external_mini_cluster.cc:1428] Setting key fc296befa53f81621eddac8746c47b68
Sep 24 00:53:00 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:00.267071 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.249107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:38523 (local address 127.0.222.62:42639)
0924 00:53:00.249307 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.249312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.250567 (+ 1255us) server_negotiation.cc:407] Connection header received
0924 00:53:00.252423 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.252429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.252531 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.252635 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:00.254404 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.255418 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.257467 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.257903 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.258000 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.261664 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:00.261669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:00.261672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:00.261706 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:00.264271 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.265073 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.265077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.265079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.265213 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.265595 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.265598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.265600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.265874 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:00.265980 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.266649 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.266836 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:53:00.268142 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.249619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:38523)
0924 00:53:00.250401 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:00.250439 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:00.252140 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:00.252871 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:00.252879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:00.253462 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:00.254199 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.254225 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.255601 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.255606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.257334 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.257350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.258122 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.258128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.258343 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.259053 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:00.259078 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:00.261414 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:00.264431 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:00.264439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:00.264455 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:00.264971 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:00.265314 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:00.265318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:00.265320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:00.265508 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:00.265987 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:00.265994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:00.266338 (+ 344us) client_negotiation.cc:769] Sending connection context
0924 00:53:00.266567 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 00:53:00.267036 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":322,"threads_started":1}
I20260924 00:53:00.269413 15392 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42639
I20260924 00:53:00.269738 15392 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:00.270500 15392 heartbeater.cc:511] Master 127.0.222.62:42639 requested a full tablet report, sending...
I20260924 00:53:00.271896 14931 ts_manager.cc:194] Registered new tserver with Master: 87844157e1374003816059043822dc21 (127.0.222.3:36981)
I20260924 00:53:00.272686 14931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:38523
I20260924 00:53:00.282644 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:53:00.300230 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.287099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.0.222.62:42639)
0924 00:53:00.287300 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.287308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.287744 (+ 436us) server_negotiation.cc:407] Connection header received
0924 00:53:00.287910 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.287915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.288003 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.288094 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:00.289640 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.290446 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.291907 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.292352 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.292477 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.295075 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:00.295080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:00.295084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:00.295118 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:00.297542 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.298282 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.298288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.298292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.298429 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.298780 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.298785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.298789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.299131 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:00.299378 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.299670 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.299890 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:53:00.304124 14931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:53:00.310927 14931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:53:00.335451 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.323687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.0.222.1:35071)
0924 00:53:00.324592 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.324601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.325520 (+ 919us) server_negotiation.cc:407] Connection header received
0924 00:53:00.325812 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.325822 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.326103 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.326283 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:00.327296 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.328981 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.331925 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.333656 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.333770 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.334045 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.334823 (+ 778us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.335037 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":1128,"threads_started":1}
I20260924 00:53:00.336489 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.323115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38240)
0924 00:53:00.325406 (+ 2291us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:00.325425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:00.325628 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:00.326611 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:00.326620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:00.326654 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:00.327072 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.327084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.329179 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.329185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.331720 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.331738 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.334273 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.334281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.334535 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.334588 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:53:00.335769 (+ 1181us) client_negotiation.cc:240] Negotiation successful
0924 00:53:00.336044 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2172,"thread_start_us":12981,"threads_started":1}
I20260924 00:53:00.340631 15056 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bd1872e5949c68fe2af372c49db40 (DEFAULT_TABLE table=test-table [id=3beca6d2abfc4db1a1ee1da778020736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:00.342177 15056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bd1872e5949c68fe2af372c49db40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:00.344391 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.323162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.0.222.3:36981)
0924 00:53:00.337240 (+ 14078us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.337248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.337280 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:53:00.337375 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.337382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.337594 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.338284 (+ 690us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:00.338484 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.339534 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.341723 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.343174 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.343309 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.343542 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.343854 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.344031 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13884,"thread_start_us":13461,"threads_started":1}
I20260924 00:53:00.344420 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.322891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36981 (local address 127.0.0.1:49944)
0924 00:53:00.323794 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:00.323832 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:00.324004 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:00.337807 (+ 13803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:00.337812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:00.337844 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:00.338131 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.338140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.339707 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.339716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.341579 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.341592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.343409 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.343416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.343627 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.343666 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 00:53:00.343791 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 00:53:00.343937 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":3,"thread_start_us":978,"threads_started":1}
I20260924 00:53:00.347023 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.325003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.0.222.2:36281)
0924 00:53:00.337079 (+ 12076us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.337089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.337114 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:53:00.337252 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.337260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.337467 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.337643 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:00.340102 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.341621 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.343923 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.345290 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.345408 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.345599 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.346514 (+ 915us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.346689 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11932,"thread_start_us":11826,"threads_started":1}
I20260924 00:53:00.347070 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.324593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:36281 (local address 127.0.0.1:54450)
0924 00:53:00.336397 (+ 11804us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:00.336418 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:00.336549 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:00.337649 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:00.337654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:00.337680 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:00.339847 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.339861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.341741 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.341745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.343718 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.343730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.346129 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.346137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.346311 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.346344 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:53:00.346443 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 00:53:00.346588 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11664,"mutex_wait_us":26,"thread_start_us":616,"threads_started":1}
I20260924 00:53:00.349308 15326 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bd1872e5949c68fe2af372c49db40 (DEFAULT_TABLE table=test-table [id=3beca6d2abfc4db1a1ee1da778020736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:00.350611 15191 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2bd1872e5949c68fe2af372c49db40 (DEFAULT_TABLE table=test-table [id=3beca6d2abfc4db1a1ee1da778020736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:00.351029 15326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bd1872e5949c68fe2af372c49db40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:00.351696 15191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2bd1872e5949c68fe2af372c49db40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:00.359726 15410 tablet_bootstrap.cc:492] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Bootstrap starting.
I20260924 00:53:00.364897 15411 tablet_bootstrap.cc:492] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Bootstrap starting.
I20260924 00:53:00.364953 15410 tablet_bootstrap.cc:654] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:00.366386 15410 log.cc:824] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:00.366941 15412 tablet_bootstrap.cc:492] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Bootstrap starting.
I20260924 00:53:00.369091 15411 tablet_bootstrap.cc:654] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:00.370251 15410 tablet_bootstrap.cc:492] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: No bootstrap required, opened a new log
I20260924 00:53:00.370369 15412 tablet_bootstrap.cc:654] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:00.370433 15411 log.cc:824] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:00.370524 15410 ts_tablet_manager.cc:1397] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 00:53:00.371589 15412 log.cc:824] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:00.373973 15411 tablet_bootstrap.cc:492] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: No bootstrap required, opened a new log
I20260924 00:53:00.374356 15411 ts_tablet_manager.cc:1397] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 00:53:00.379309 15410 raft_consensus.cc:348] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.379683 15410 raft_consensus.cc:374] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:00.379786 15410 raft_consensus.cc:729] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ca92cd60d9414cbf8c933e8de1065a, State: Initialized, Role: FOLLOWER
I20260924 00:53:00.380329 15412 tablet_bootstrap.cc:492] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: No bootstrap required, opened a new log
I20260924 00:53:00.380388 15410 consensus_queue.cc:260] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.380632 15412 ts_tablet_manager.cc:1397] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260924 00:53:00.382050 15410 ts_tablet_manager.cc:1428] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260924 00:53:00.385543 15411 raft_consensus.cc:348] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.386029 15411 raft_consensus.cc:374] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:00.386133 15411 raft_consensus.cc:729] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7870ec61274e42e9b3c685d708fe9045, State: Initialized, Role: FOLLOWER
I20260924 00:53:00.386831 15411 consensus_queue.cc:260] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.388641 15257 heartbeater.cc:503] Master 127.0.222.62:42639 was elected leader, sending a full tablet report...
I20260924 00:53:00.389122 15411 ts_tablet_manager.cc:1428] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260924 00:53:00.390573 15412 raft_consensus.cc:348] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.391068 15412 raft_consensus.cc:374] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:00.391170 15412 raft_consensus.cc:729] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87844157e1374003816059043822dc21, State: Initialized, Role: FOLLOWER
I20260924 00:53:00.391685 15412 consensus_queue.cc:260] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.392769 15392 heartbeater.cc:503] Master 127.0.222.62:42639 was elected leader, sending a full tablet report...
I20260924 00:53:00.393509 15412 ts_tablet_manager.cc:1428] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260924 00:53:00.460707 15417 raft_consensus.cc:482] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:53:00.461030 15417 raft_consensus.cc:504] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.462708 15417 leader_election.cc:288] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87844157e1374003816059043822dc21 (127.0.222.3:36981), c6ca92cd60d9414cbf8c933e8de1065a (127.0.222.1:35071)
I20260924 00:53:00.469141 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.463164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:38701 (local address 127.0.222.3:36981)
0924 00:53:00.463379 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.463386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.463873 (+ 487us) server_negotiation.cc:407] Connection header received
0924 00:53:00.464064 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.464068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.464133 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.464214 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:00.464721 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.465490 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.467259 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.468066 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.468183 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.468287 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.468753 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.468885 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 00:53:00.469159 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.463173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:38765 (local address 127.0.222.1:35071)
0924 00:53:00.463569 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.463575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.464138 (+ 563us) server_negotiation.cc:407] Connection header received
0924 00:53:00.464309 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.464314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.464370 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.464443 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:00.464964 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.465684 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.467401 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.468194 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.468315 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.468416 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.468796 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.468919 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260924 00:53:00.469375 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.463019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.222.2:38765)
0924 00:53:00.464045 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:00.464058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:00.464168 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:00.464486 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:00.464490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:00.464517 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:00.464828 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.464836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.465823 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.465827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.467252 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.467264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.468417 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.468424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.468642 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.468669 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 00:53:00.468746 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:53:00.468875 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":51,"thread_start_us":390,"threads_started":1}
I20260924 00:53:00.469581 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.463003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36981 (local address 127.0.222.2:38701)
0924 00:53:00.463757 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:00.463773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:00.463937 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:00.464230 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:00.464234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:00.464258 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:00.464584 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.464593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.465607 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.465612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.467101 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:00.467114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.468280 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.468287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:00.468543 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.468607 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 00:53:00.468695 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:53:00.468836 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":88,"thread_start_us":752,"threads_started":1}
I20260924 00:53:00.470467 15076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bd1872e5949c68fe2af372c49db40" candidate_uuid: "7870ec61274e42e9b3c685d708fe9045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" is_pre_election: true
I20260924 00:53:00.470516 15346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bd1872e5949c68fe2af372c49db40" candidate_uuid: "7870ec61274e42e9b3c685d708fe9045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87844157e1374003816059043822dc21" is_pre_election: true
I20260924 00:53:00.471088 15346 raft_consensus.cc:2454] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7870ec61274e42e9b3c685d708fe9045 in term 0.
I20260924 00:53:00.471088 15076 raft_consensus.cc:2454] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7870ec61274e42e9b3c685d708fe9045 in term 0.
I20260924 00:53:00.471796 15146 leader_election.cc:302] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7870ec61274e42e9b3c685d708fe9045, c6ca92cd60d9414cbf8c933e8de1065a; no voters:
I20260924 00:53:00.472216 15417 raft_consensus.cc:2781] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:53:00.472396 15417 raft_consensus.cc:482] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:53:00.472509 15417 raft_consensus.cc:3037] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:53:00.473912 15123 tablet.cc:2369] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:53:00.474951 15417 raft_consensus.cc:504] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.475620 15417 leader_election.cc:288] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [CANDIDATE]: Term 1 election: Requested vote from peers 87844157e1374003816059043822dc21 (127.0.222.3:36981), c6ca92cd60d9414cbf8c933e8de1065a (127.0.222.1:35071)
I20260924 00:53:00.476218 15346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bd1872e5949c68fe2af372c49db40" candidate_uuid: "7870ec61274e42e9b3c685d708fe9045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87844157e1374003816059043822dc21"
I20260924 00:53:00.476291 15076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2bd1872e5949c68fe2af372c49db40" candidate_uuid: "7870ec61274e42e9b3c685d708fe9045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ca92cd60d9414cbf8c933e8de1065a"
I20260924 00:53:00.476445 15346 raft_consensus.cc:3037] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:00.476480 15076 raft_consensus.cc:3037] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:00.479043 15346 raft_consensus.cc:2454] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7870ec61274e42e9b3c685d708fe9045 in term 1.
I20260924 00:53:00.479056 15076 raft_consensus.cc:2454] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7870ec61274e42e9b3c685d708fe9045 in term 1.
I20260924 00:53:00.479452 15147 leader_election.cc:302] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7870ec61274e42e9b3c685d708fe9045, 87844157e1374003816059043822dc21; no voters:
I20260924 00:53:00.479773 15417 raft_consensus.cc:2781] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:53:00.480271 15417 raft_consensus.cc:686] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 1 LEADER]: Becoming Leader. State: Replica: 7870ec61274e42e9b3c685d708fe9045, State: Running, Role: LEADER
I20260924 00:53:00.480736 15417 consensus_queue.cc:237] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } }
I20260924 00:53:00.485842 14931 catalog_manager.cc:5495] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2). New cstate: current_term: 1 leader_uuid: "7870ec61274e42e9b3c685d708fe9045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } health_report { overall_health: HEALTHY } } }
W20260924 00:53:00.494310 15258 tablet.cc:2369] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:53:00.498190 15393 tablet.cc:2369] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:53:00.527531 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.522176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.0.222.2:36281)
0924 00:53:00.522382 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.522386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.522404 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:53:00.522497 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.522501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.522579 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.522698 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:53:00.523196 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.523948 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.525459 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.525851 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.525951 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.526426 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:53:00.526962 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:53:00.527072 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.527185 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.527256 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:53:00.535662 15076 raft_consensus.cc:1261] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Refusing update from remote peer 7870ec61274e42e9b3c685d708fe9045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:53:00.535679 15346 raft_consensus.cc:1261] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 1 FOLLOWER]: Refusing update from remote peer 7870ec61274e42e9b3c685d708fe9045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:53:00.536612 15421 consensus_queue.cc:1035] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:53:00.536927 15417 consensus_queue.cc:1035] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:53:00.548753 15424 mvcc.cc:205] Tried to move back new op lower bound from 7332704995464437760 to 7332704995253534720. Current Snapshot: MvccSnapshot[applied={T|T < 7332704995464437760}]
I20260924 00:53:00.554694 15426 mvcc.cc:205] Tried to move back new op lower bound from 7332704995464437760 to 7332704995253534720. Current Snapshot: MvccSnapshot[applied={T|T < 7332704995464437760}]
I20260924 00:53:00.565570 14931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:53:00.565908 14931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:53:00.569895 14931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Sending DeleteTablet for 3 replicas of tablet 0d2bd1872e5949c68fe2af372c49db40
I20260924 00:53:00.571175 15326 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bd1872e5949c68fe2af372c49db40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:00 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49944
I20260924 00:53:00.571244 15056 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bd1872e5949c68fe2af372c49db40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:00 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:38240
I20260924 00:53:00.571444 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 14993
I20260924 00:53:00.574419 15191 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2bd1872e5949c68fe2af372c49db40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:00 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:54450
I20260924 00:53:00.582685 15434 tablet_replica.cc:330] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: stopping tablet replica
I20260924 00:53:00.582762 15435 tablet_replica.cc:330] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: stopping tablet replica
I20260924 00:53:00.583124 15434 raft_consensus.cc:2229] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:53:00.583232 15435 raft_consensus.cc:2229] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:53:00.583658 15434 raft_consensus.cc:2258] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:00.583707 15435 raft_consensus.cc:2258] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:00.584085 15437 tablet_replica.cc:330] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: stopping tablet replica
I20260924 00:53:00.584525 15437 raft_consensus.cc:2229] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:53:00.585289 15437 raft_consensus.cc:2258] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:00.586627 15435 ts_tablet_manager.cc:1905] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:00.586943 15434 ts_tablet_manager.cc:1905] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:00.588481 15437 ts_tablet_manager.cc:1905] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:00.594244 15434 ts_tablet_manager.cc:1918] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:00.594434 15434 log.cc:1196] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/0d2bd1872e5949c68fe2af372c49db40
I20260924 00:53:00.595007 15434 ts_tablet_manager.cc:1939] T 0d2bd1872e5949c68fe2af372c49db40 P 87844157e1374003816059043822dc21: Deleting consensus metadata
Sep 24 00:53:00 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:53:00.596117 15435 ts_tablet_manager.cc:1918] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:00.596429 15435 log.cc:1196] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/0d2bd1872e5949c68fe2af372c49db40
I20260924 00:53:00.597047 15435 ts_tablet_manager.cc:1939] T 0d2bd1872e5949c68fe2af372c49db40 P c6ca92cd60d9414cbf8c933e8de1065a: Deleting consensus metadata
I20260924 00:53:00.597064 14918 catalog_manager.cc:4841] TS 87844157e1374003816059043822dc21 (127.0.222.3:36981): tablet 0d2bd1872e5949c68fe2af372c49db40 (table test-table [id=3beca6d2abfc4db1a1ee1da778020736]) successfully deleted
I20260924 00:53:00.599237 14917 catalog_manager.cc:4841] TS c6ca92cd60d9414cbf8c933e8de1065a (127.0.222.1:35071): tablet 0d2bd1872e5949c68fe2af372c49db40 (table test-table [id=3beca6d2abfc4db1a1ee1da778020736]) successfully deleted
I20260924 00:53:00.603694 15437 ts_tablet_manager.cc:1918] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:00.603897 15437 log.cc:1196] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/0d2bd1872e5949c68fe2af372c49db40
I20260924 00:53:00.604529 15437 ts_tablet_manager.cc:1939] T 0d2bd1872e5949c68fe2af372c49db40 P 7870ec61274e42e9b3c685d708fe9045: Deleting consensus metadata
I20260924 00:53:00.605160 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.572559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.0.222.1:35071)
0924 00:53:00.572754 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.572761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.584387 (+ 11626us) server_negotiation.cc:407] Connection header received
0924 00:53:00.585138 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.585144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.585249 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.585360 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:00.587318 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.588417 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.590506 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.591168 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.591267 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.597642 (+ 6375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:00.597658 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:00.597677 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:00.597721 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:00.601613 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.602615 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.602625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.602630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.602821 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.603347 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.603353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.603356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.603813 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:00.604115 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.604569 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.604791 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:53:00.605950 15117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:00.606539 14918 catalog_manager.cc:4841] TS 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281): tablet 0d2bd1872e5949c68fe2af372c49db40 (table test-table [id=3beca6d2abfc4db1a1ee1da778020736]) successfully deleted
I20260924 00:53:00.715158 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 14993
I20260924 00:53:00.729183 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 15128
Sep 24 00:53:00 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:53:00.751091 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.730002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.0.222.2:36281)
0924 00:53:00.730200 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.730215 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.730237 (+ 22us) server_negotiation.cc:407] Connection header received
0924 00:53:00.730358 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.730363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.730446 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.730558 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:00.731872 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.733095 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.734538 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.735237 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.735376 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.744387 (+ 9011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:00.744401 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:00.744421 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:00.744473 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:00.747733 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.748522 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.748530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.748532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.748694 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.749344 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.749349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.749353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.749752 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:00.750096 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.750424 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.750647 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 00:53:00.751767 15252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:00.872200 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 15128
I20260924 00:53:00.894466 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 15263
Sep 24 00:53:00 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:53:00.921679 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:00.895204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.0.222.3:36981)
0924 00:53:00.895395 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:00.895402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:00.895560 (+ 158us) server_negotiation.cc:407] Connection header received
0924 00:53:00.895805 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:00.895813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:00.895916 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:00.895995 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:00.905319 (+ 9324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.906291 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.908359 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:00.909026 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:00.909179 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:00.915120 (+ 5941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:00.915141 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:00.915159 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:00.915208 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:00.918209 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.919060 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.919069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.919071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.919326 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:00.919879 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:00.919883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:00.919885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:00.920296 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:00.920605 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:00.920986 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 00:53:00.921267 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:53:00.922399 15387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:01.012401 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 15263
I20260924 00:53:01.026296 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 14899
I20260924 00:53:01.027410 14962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:01.136139 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 14899
I20260924 00:53:01.153532 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42639
--webserver_interface=127.0.222.62
--webserver_port=33927
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:01.492089 15442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:01.492409 15442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:01.492498 15442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:01.499879 15442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:53:01.500005 15442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:01.500049 15442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:01.500103 15442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:53:01.500145 15442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:53:01.510129 15442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=33927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:01.511456 15442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:01.513068 15442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:01.541740 15448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:01.541771 15451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:01.541716 15449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:01.542680 15442 server_base.cc:1034] running on GCE node
I20260924 00:53:01.543582 15442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:01.545342 15442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:01.546566 15442 hybrid_clock.cc:648] HybridClock initialized: now 1790211181546493 us; error 70 us; skew 500 ppm
I20260924 00:53:01.546988 15442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:01 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:01.550678 15442 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:53:01.552829 15442 webserver.cc:460] Webserver started at http://127.0.222.62:33927/ using document root <none> and password file <none>
I20260924 00:53:01.553757 15442 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:01.553995 15442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:01.560822 15442 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 00:53:01.564525 15458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:01.566390 15442 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 00:53:01.566601 15442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "04ce98d104fb4061b530edbae7654c03"
format_stamp: "Formatted at 2026-09-24 00:52:58 on dist-test-slave-rtvz"
server_key: "283d35ab61044792c2498271c6283de7"
server_key_iv: "bb7517ea2a82cd6239379bf791b161c2"
server_key_version: "encryptionkey@0"
I20260924 00:53:01.567828 15442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:01.591491 15442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:01.596619 15442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:01.596951 15442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:01.615979 15442 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42639
I20260924 00:53:01.615998 15509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42639 every 8 connection(s)
I20260924 00:53:01.618654 15442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:53:01.621142 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 15442
I20260924 00:53:01.622009 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:35071
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:42639
--webserver_port=37897
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:53:01.626966 15510 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:53:01.630609 15510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Bootstrap starting.
I20260924 00:53:01.650519 15510 log.cc:824] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:01.663048 15510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:53:01.663580 15510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Bootstrap complete.
I20260924 00:53:01.673015 15510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:53:01.673974 15510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ce98d104fb4061b530edbae7654c03, State: Initialized, Role: FOLLOWER
I20260924 00:53:01.674640 15510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:53:01.674909 15510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:53:01.675029 15510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:53:01.675205 15510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:53:01.679349 15510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:53:01.679927 15510 leader_election.cc:302] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04ce98d104fb4061b530edbae7654c03; no voters:
I20260924 00:53:01.680593 15510 leader_election.cc:288] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:53:01.680737 15514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:53:01.681931 15514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 2 LEADER]: Becoming Leader. State: Replica: 04ce98d104fb4061b530edbae7654c03, State: Running, Role: LEADER
I20260924 00:53:01.682575 15514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:53:01.682940 15510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:53:01.691661 15516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04ce98d104fb4061b530edbae7654c03. Latest consensus state: current_term: 2 leader_uuid: "04ce98d104fb4061b530edbae7654c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } } }
I20260924 00:53:01.692163 15516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: This master's current role is: LEADER
I20260924 00:53:01.701530 15515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "04ce98d104fb4061b530edbae7654c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } } }
I20260924 00:53:01.701930 15515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: This master's current role is: LEADER
I20260924 00:53:01.702665 15524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:53:01.711946 15524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:53:01.713795 15524 catalog_manager.cc:1259] Loaded cluster ID: 4f797f09e0094f91a4e7dc4864cc0467
I20260924 00:53:01.713933 15524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:53:01.717254 15524 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 00:53:01.734536 15524 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 00:53:01.744432 15524 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:53:01.746206 15524 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Loaded TSK: 0
I20260924 00:53:01.747848 15524 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:53:01.984220 15512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:01.984565 15512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:01.984714 15512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:01.993834 15512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:01.993954 15512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:01.994117 15512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:53:02.004575 15512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:35071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=37897
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:02.005998 15512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:02.007391 15512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:02.022781 15539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:02.022939 15541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:02.023124 15538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:02.024704 15512 server_base.cc:1034] running on GCE node
I20260924 00:53:02.025437 15512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:02.026597 15512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:02.027837 15512 hybrid_clock.cc:648] HybridClock initialized: now 1790211182027769 us; error 66 us; skew 500 ppm
I20260924 00:53:02.028159 15512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:02 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:02.031967 15512 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:53:02.033993 15512 webserver.cc:460] Webserver started at http://127.0.222.1:37897/ using document root <none> and password file <none>
I20260924 00:53:02.034850 15512 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:02.034988 15512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:02.041967 15512 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 00:53:02.045521 15548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:02.047309 15512 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 00:53:02.047518 15512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "c6ca92cd60d9414cbf8c933e8de1065a"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "4716b35ec88ff7a67d072d7922c81ec7"
server_key_iv: "505c042b520c70ba649313711594cef0"
server_key_version: "encryptionkey@0"
I20260924 00:53:02.049001 15512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:02.079389 15512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:02.084818 15512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:02.085163 15512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:02.086374 15512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:02.088096 15512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:02.088202 15512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:02.088313 15512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:02.088382 15512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:02.120652 15512 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:35071
I20260924 00:53:02.120718 15661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:35071 every 8 connection(s)
I20260924 00:53:02.122217 15512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:53:02.126052 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 15512
I20260924 00:53:02.126703 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:36281
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:42639
--webserver_port=33975
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:53:02 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:02.149330 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:02.125147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:38515 (local address 127.0.222.62:42639)
0924 00:53:02.125906 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:02.125923 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:02.126538 (+ 615us) server_negotiation.cc:407] Connection header received
0924 00:53:02.128823 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:02.128843 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:02.129360 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:02.129767 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:02.131336 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:02.133081 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:02.135469 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:02.136209 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:02.136391 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:02.141011 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:02.141021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:02.141038 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:02.141089 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:02.144226 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:02.145384 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:02.145393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:02.145399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:02.145596 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:02.146160 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:02.146164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:02.146166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:02.146702 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:02.146918 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:02.147953 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0924 00:53:02.148310 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":297,"threads_started":1}
I20260924 00:53:02.149643 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:02.125456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:38515)
0924 00:53:02.126334 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:02.126388 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:02.128191 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:02.129972 (+ 1781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:02.129980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:02.130526 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:02.131102 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:02.131121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:02.133307 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:02.133317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:02.135290 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:02.135308 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:02.136453 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:02.136468 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:02.136759 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:02.137599 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:02.137630 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:02.140520 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:02.144438 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:02.144450 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:02.144470 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:02.145166 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:02.145724 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:02.145729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:02.145733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:02.146017 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:02.146851 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:02.146861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:02.147320 (+ 459us) client_negotiation.cc:769] Sending connection context
0924 00:53:02.147658 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 00:53:02.148282 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":353,"threads_started":1}
I20260924 00:53:02.153558 15662 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42639
I20260924 00:53:02.153980 15662 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:02.154920 15662 heartbeater.cc:511] Master 127.0.222.62:42639 requested a full tablet report, sending...
I20260924 00:53:02.157449 15475 ts_manager.cc:194] Registered new tserver with Master: c6ca92cd60d9414cbf8c933e8de1065a (127.0.222.1:35071)
I20260924 00:53:02.159024 15475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:38515
W20260924 00:53:02.474794 15666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:02.475102 15666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:02.475225 15666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:02.482832 15666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:02.482951 15666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:02.483098 15666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:53:02.493549 15666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:36281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=33975
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:02.494937 15666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:02.496290 15666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:02.509888 15672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:02.509934 15671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:02.509950 15674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:02.518894 15666 server_base.cc:1034] running on GCE node
I20260924 00:53:02.519614 15666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:02.520637 15666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:02.521873 15666 hybrid_clock.cc:648] HybridClock initialized: now 1790211182521824 us; error 36 us; skew 500 ppm
I20260924 00:53:02.522200 15666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:02 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:02.525447 15666 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:53:02.527077 15666 webserver.cc:460] Webserver started at http://127.0.222.2:33975/ using document root <none> and password file <none>
I20260924 00:53:02.527842 15666 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:02.527985 15666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:02.533906 15666 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 00:53:02.536749 15681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:02.538285 15666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:53:02.538448 15666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "7870ec61274e42e9b3c685d708fe9045"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "598817166d4143008ebe1b11aeaf3ae5"
server_key_iv: "1f1d589ca979439a3bb898555763b54c"
server_key_version: "encryptionkey@0"
I20260924 00:53:02.539470 15666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:02.560850 15666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:02.565337 15666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:02.565599 15666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:02.566586 15666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:02.568076 15666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:02.568172 15666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:02.568293 15666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:02.568365 15666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:02.600301 15666 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:36281
I20260924 00:53:02.600358 15794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:36281 every 8 connection(s)
I20260924 00:53:02.602183 15666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:53:02.602948 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 15666
I20260924 00:53:02.603744 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:36981
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:42639
--webserver_port=43049
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:53:02 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:02.627432 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:02.605136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:57799 (local address 127.0.222.62:42639)
0924 00:53:02.605379 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:02.605386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:02.606856 (+ 1470us) server_negotiation.cc:407] Connection header received
0924 00:53:02.609440 (+ 2584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:02.609445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:02.609526 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:02.609617 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:02.611744 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:02.612844 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:02.614643 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:02.615276 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:02.615427 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:02.619954 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:02.619961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:02.619967 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:02.620009 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:02.623509 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:02.624571 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:02.624577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:02.624581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:02.624754 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:02.625346 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:02.625352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:02.625356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:02.625703 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:02.625837 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:02.626801 (+ 964us) server_negotiation.cc:299] Negotiation successful
0924 00:53:02.627056 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 00:53:02.628845 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:02.605736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:57799)
0924 00:53:02.606646 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:02.606710 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:02.609037 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:02.609933 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:02.609945 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:02.610680 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:02.611504 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:02.611526 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:02.613029 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:02.613033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:02.614479 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:02.614494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:02.615564 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:02.615572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:02.615882 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:02.616684 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:02.616721 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:02.619693 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:02.623702 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:02.623713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:02.623735 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:02.624418 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:02.624888 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:02.624894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:02.624897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:02.625178 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:02.625836 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:02.625845 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:02.626355 (+ 510us) client_negotiation.cc:769] Sending connection context
0924 00:53:02.626694 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 00:53:02.627342 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":338,"threads_started":1}
I20260924 00:53:02.630453 15795 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42639
I20260924 00:53:02.630787 15795 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:02.631670 15795 heartbeater.cc:511] Master 127.0.222.62:42639 requested a full tablet report, sending...
I20260924 00:53:02.633407 15475 ts_manager.cc:194] Registered new tserver with Master: 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281)
I20260924 00:53:02.634516 15475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:57799
W20260924 00:53:02.938870 15798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:02.939179 15798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:02.939332 15798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:02.946841 15798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:02.946986 15798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:02.947150 15798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:53:02.957747 15798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:36981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=43049
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:02.959134 15798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:02.960513 15798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:02.974298 15803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:02.974310 15804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:02.974479 15806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:02.975239 15798 server_base.cc:1034] running on GCE node
I20260924 00:53:02.975924 15798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:02.976967 15798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:02.978132 15798 hybrid_clock.cc:648] HybridClock initialized: now 1790211182978067 us; error 60 us; skew 500 ppm
I20260924 00:53:02.978451 15798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:02 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:02.981917 15798 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:53:02.983506 15798 webserver.cc:460] Webserver started at http://127.0.222.3:43049/ using document root <none> and password file <none>
I20260924 00:53:02.984282 15798 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:02.984408 15798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:02.990206 15798 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 00:53:02.993067 15813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:02.994928 15798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 00:53:02.995100 15798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "87844157e1374003816059043822dc21"
format_stamp: "Formatted at 2026-09-24 00:53:00 on dist-test-slave-rtvz"
server_key: "d60341c58f15ab4834f786ad6cee5142"
server_key_iv: "68eb2c54385b1aec6e5cce86b2fcae6d"
server_key_version: "encryptionkey@0"
I20260924 00:53:02.996332 15798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:03.007810 15798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:03.012243 15798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:03.012490 15798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:03.013733 15798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:03.015266 15798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:03.015369 15798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:03.015453 15798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:03.015506 15798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:03.046446 15798 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:36981
I20260924 00:53:03.046505 15926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:36981 every 8 connection(s)
I20260924 00:53:03.047889 15798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:53:03.049729 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 15798
Sep 24 00:53:03 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:03.066972 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.050805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.222.62:42639)
0924 00:53:03.051904 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.051911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.051934 (+ 23us) server_negotiation.cc:407] Connection header received
0924 00:53:03.052004 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.052009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.052089 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.052213 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:03.054031 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.055131 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.056815 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.057335 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.057438 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.060578 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:03.060585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:03.060590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:03.060632 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:03.063868 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.064782 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.064786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.064788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.064930 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.065420 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.065424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.065427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.065795 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:03.066055 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.066420 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.066647 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946}
I20260924 00:53:03.070942 15930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.050805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:51849 (local address 127.0.222.62:42639)
0924 00:53:03.053521 (+ 2716us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.053528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.053548 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:53:03.054895 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.054899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.054969 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.055079 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:03.056717 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.057746 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.059451 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.060192 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.060334 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.064898 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:03.064904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:03.064907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:03.064942 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:03.067848 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.068644 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.068649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.068652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.068808 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.069241 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.069246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.069248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.069600 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:03.069727 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.070453 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.070650 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":2586,"thread_start_us":2427,"threads_started":1}
I20260924 00:53:03.072033 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.051466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:51849)
0924 00:53:03.052264 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.052312 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:03.054582 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:03.055352 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:03.055365 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:03.055907 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:03.056511 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.056530 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.057905 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.057909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.059285 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.059305 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.060439 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.060449 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.060911 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.061821 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:03.061857 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:03.064652 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:03.067992 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:03.068003 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:03.068020 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:03.068525 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:03.068918 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:03.068921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:03.068923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:03.069115 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:03.069735 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:03.069743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:03.070080 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 00:53:03.070360 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 00:53:03.070813 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":769,"threads_started":1}
I20260924 00:53:03.073556 15927 heartbeater.cc:347] Connected to a master server at 127.0.222.62:42639
I20260924 00:53:03.073864 15927 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:03.074622 15927 heartbeater.cc:511] Master 127.0.222.62:42639 requested a full tablet report, sending...
I20260924 00:53:03.076146 15475 ts_manager.cc:194] Registered new tserver with Master: 87844157e1374003816059043822dc21 (127.0.222.3:36981)
I20260924 00:53:03.076978 15475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:51849
I20260924 00:53:03.082226 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:53:03.089097 15930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.083119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.0.222.62:42639)
0924 00:53:03.083277 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.083282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.083718 (+ 436us) server_negotiation.cc:407] Connection header received
0924 00:53:03.083918 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.083922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.084109 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.084226 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:53:03.084736 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.085651 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.087158 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.087600 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.087692 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.088131 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:53:03.088631 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:53:03.088739 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.088828 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.088897 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:53:03.090041 15475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:53:03.097229 15475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:53:03.121008 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.109868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38262)
0924 00:53:03.110961 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.110983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:03.111113 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:03.114399 (+ 3286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:03.114405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:03.114434 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:03.114825 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.114838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.116428 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.116433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.118095 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.118107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.119911 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.119918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.120215 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.120240 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 00:53:03.120288 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:53:03.120508 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":84,"thread_start_us":846,"threads_started":1}
I20260924 00:53:03.121064 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.109121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:36281 (local address 127.0.0.1:54468)
0924 00:53:03.109990 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.110042 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:03.110285 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:03.112483 (+ 2198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:03.112492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:03.112531 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:03.113027 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.113042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.115354 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.115360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.118041 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.118061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.119930 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.119936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.120114 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.120175 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 00:53:03.120285 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 00:53:03.120508 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":103,"thread_start_us":413,"threads_started":1}
I20260924 00:53:03.121636 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.109472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.0.222.2:36281)
0924 00:53:03.111694 (+ 2222us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.111706 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.111749 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:53:03.111874 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.111883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.112189 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.113588 (+ 1399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:03.113887 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.115149 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.118224 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.119705 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.120794 (+ 1089us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.121031 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.121119 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.121313 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2021,"thread_start_us":1897,"threads_started":1}
I20260924 00:53:03.122361 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.110504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.222.1:35071)
0924 00:53:03.113862 (+ 3358us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.113870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.113897 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:53:03.113983 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.113990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.114202 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.114391 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:03.115002 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.116311 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.118264 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.119676 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.121526 (+ 1850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.121820 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.121874 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.121991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3188,"thread_start_us":3044,"threads_started":1}
I20260924 00:53:03.122637 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.111098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.0.222.3:36981)
0924 00:53:03.112014 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.112023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.112728 (+ 705us) server_negotiation.cc:407] Connection header received
0924 00:53:03.112826 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.112834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.113039 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.113215 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:03.113790 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.115148 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.118633 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.119601 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.119772 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.120042 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.121221 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.122259 (+ 1038us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":603,"threads_started":1}
I20260924 00:53:03.123212 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.110461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36981 (local address 127.0.0.1:49958)
0924 00:53:03.112463 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.112479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:03.112598 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:03.113221 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:03.113225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:03.113248 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:03.113579 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.113591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.115290 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.115296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.118480 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.118491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.120224 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.120231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.120413 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.120448 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:53:03.120541 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 00:53:03.120692 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1886,"mutex_wait_us":52,"thread_start_us":1749,"threads_started":1}
I20260924 00:53:03.125937 15730 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063 (DEFAULT_TABLE table=test-table [id=04d6bda70f8845c497a017cdbf261858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:03.125955 15596 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063 (DEFAULT_TABLE table=test-table [id=04d6bda70f8845c497a017cdbf261858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:03.127378 15730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:03.127468 15596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:03.129942 15862 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063 (DEFAULT_TABLE table=test-table [id=04d6bda70f8845c497a017cdbf261858]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:03.131289 15862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:03.144810 15945 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Bootstrap starting.
I20260924 00:53:03.146944 15943 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Bootstrap starting.
I20260924 00:53:03.147734 15944 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Bootstrap starting.
I20260924 00:53:03.148593 15945 tablet_bootstrap.cc:654] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:03.149982 15945 log.cc:824] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:03.150857 15943 tablet_bootstrap.cc:654] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:03.151170 15944 tablet_bootstrap.cc:654] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:03.152053 15943 log.cc:824] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:03.152554 15944 log.cc:824] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:03.154896 15943 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: No bootstrap required, opened a new log
I20260924 00:53:03.155181 15943 ts_tablet_manager.cc:1397] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 00:53:03.155305 15944 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: No bootstrap required, opened a new log
I20260924 00:53:03.155668 15944 ts_tablet_manager.cc:1397] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 00:53:03.156701 15945 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: No bootstrap required, opened a new log
I20260924 00:53:03.156981 15945 ts_tablet_manager.cc:1397] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 00:53:03.163075 15662 heartbeater.cc:503] Master 127.0.222.62:42639 was elected leader, sending a full tablet report...
I20260924 00:53:03.163235 15943 raft_consensus.cc:348] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.163693 15943 raft_consensus.cc:374] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:03.163789 15943 raft_consensus.cc:729] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7870ec61274e42e9b3c685d708fe9045, State: Initialized, Role: FOLLOWER
I20260924 00:53:03.164649 15943 consensus_queue.cc:260] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.166262 15944 raft_consensus.cc:348] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.166762 15944 raft_consensus.cc:374] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:03.166908 15944 raft_consensus.cc:729] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ca92cd60d9414cbf8c933e8de1065a, State: Initialized, Role: FOLLOWER
I20260924 00:53:03.167364 15945 raft_consensus.cc:348] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.167618 15944 consensus_queue.cc:260] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.167896 15945 raft_consensus.cc:374] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:03.168043 15945 raft_consensus.cc:729] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87844157e1374003816059043822dc21, State: Initialized, Role: FOLLOWER
I20260924 00:53:03.168809 15945 consensus_queue.cc:260] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.170842 15945 ts_tablet_manager.cc:1428] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 00:53:03.171677 15927 heartbeater.cc:503] Master 127.0.222.62:42639 was elected leader, sending a full tablet report...
I20260924 00:53:03.172233 15795 heartbeater.cc:503] Master 127.0.222.62:42639 was elected leader, sending a full tablet report...
I20260924 00:53:03.172767 15944 ts_tablet_manager.cc:1428] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260924 00:53:03.174185 15943 ts_tablet_manager.cc:1428] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
W20260924 00:53:03.299561 15928 tablet.cc:2369] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:53:03.306352 15950 raft_consensus.cc:482] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:53:03.306624 15950 raft_consensus.cc:504] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.308354 15950 leader_election.cc:288] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281), 87844157e1374003816059043822dc21 (127.0.222.3:36981)
I20260924 00:53:03.315289 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.308775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:55259 (local address 127.0.222.2:36281)
0924 00:53:03.308983 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.308987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.309311 (+ 324us) server_negotiation.cc:407] Connection header received
0924 00:53:03.309521 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.309526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.309605 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.309712 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:03.310331 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.311070 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.313264 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.314325 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.314421 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.314499 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.314888 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.314992 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 00:53:03.315496 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.308577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:36281 (local address 127.0.222.1:55259)
0924 00:53:03.309185 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.309234 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:03.309379 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:03.309763 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:03.309773 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:03.309804 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:03.310174 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.310183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.311195 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.311200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.313069 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.313081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.314536 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.314542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.314715 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.314752 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 00:53:03.314849 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 00:53:03.314987 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":55,"thread_start_us":307,"threads_started":1}
I20260924 00:53:03.316010 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.309456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:43197 (local address 127.0.222.3:36981)
0924 00:53:03.309671 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.309677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.310088 (+ 411us) server_negotiation.cc:407] Connection header received
0924 00:53:03.310273 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.310276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.310332 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.310401 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:03.310900 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.311838 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.313983 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.314981 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.315094 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.315193 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.315584 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.315831 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:53:03.316671 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.309346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36981 (local address 127.0.222.1:43197)
0924 00:53:03.309993 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.310007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:03.310136 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:03.310467 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:03.310472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:03.310494 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:03.310757 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.310764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.312017 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.312021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.313798 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:03.313813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.315207 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.315213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:03.315392 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.315429 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 00:53:03.316183 (+ 754us) client_negotiation.cc:240] Negotiation successful
0924 00:53:03.316323 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":4,"thread_start_us":283,"threads_started":1}
I20260924 00:53:03.316660 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd0edf6d3864ac5a8ef8b7ad1453063" candidate_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7870ec61274e42e9b3c685d708fe9045" is_pre_election: true
I20260924 00:53:03.317226 15750 raft_consensus.cc:2454] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ca92cd60d9414cbf8c933e8de1065a in term 0.
I20260924 00:53:03.317624 15882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd0edf6d3864ac5a8ef8b7ad1453063" candidate_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87844157e1374003816059043822dc21" is_pre_election: true
I20260924 00:53:03.317859 15552 leader_election.cc:302] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7870ec61274e42e9b3c685d708fe9045, c6ca92cd60d9414cbf8c933e8de1065a; no voters:
I20260924 00:53:03.318138 15882 raft_consensus.cc:2454] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ca92cd60d9414cbf8c933e8de1065a in term 0.
I20260924 00:53:03.318236 15950 raft_consensus.cc:2781] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:53:03.318373 15950 raft_consensus.cc:482] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:53:03.318456 15950 raft_consensus.cc:3037] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:03.320600 15950 raft_consensus.cc:504] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.321149 15950 leader_election.cc:288] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 1 election: Requested vote from peers 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281), 87844157e1374003816059043822dc21 (127.0.222.3:36981)
I20260924 00:53:03.321635 15750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd0edf6d3864ac5a8ef8b7ad1453063" candidate_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7870ec61274e42e9b3c685d708fe9045"
I20260924 00:53:03.321719 15882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd0edf6d3864ac5a8ef8b7ad1453063" candidate_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87844157e1374003816059043822dc21"
I20260924 00:53:03.321826 15750 raft_consensus.cc:3037] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:03.321928 15882 raft_consensus.cc:3037] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:03.324043 15750 raft_consensus.cc:2454] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ca92cd60d9414cbf8c933e8de1065a in term 1.
I20260924 00:53:03.324045 15882 raft_consensus.cc:2454] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ca92cd60d9414cbf8c933e8de1065a in term 1.
I20260924 00:53:03.324393 15552 leader_election.cc:302] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7870ec61274e42e9b3c685d708fe9045, c6ca92cd60d9414cbf8c933e8de1065a; no voters:
I20260924 00:53:03.324687 15950 raft_consensus.cc:2781] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:53:03.325093 15950 raft_consensus.cc:686] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 LEADER]: Becoming Leader. State: Replica: c6ca92cd60d9414cbf8c933e8de1065a, State: Running, Role: LEADER
I20260924 00:53:03.325608 15950 consensus_queue.cc:237] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:03.330467 15473 catalog_manager.cc:5495] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ca92cd60d9414cbf8c933e8de1065a (127.0.222.1). New cstate: current_term: 1 leader_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } health_report { overall_health: HEALTHY } } }
W20260924 00:53:03.354305 15796 tablet.cc:2369] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:53:03.370548 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.364958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.0.222.1:35071)
0924 00:53:03.365156 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.365161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.365230 (+ 69us) server_negotiation.cc:407] Connection header received
0924 00:53:03.365468 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.365472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.365559 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.365678 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:53:03.366041 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.366799 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.368259 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.368683 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.368807 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.369410 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:53:03.369947 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:53:03.370083 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.370189 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.370277 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 00:53:03.374751 15663 tablet.cc:2369] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:53:03.379151 15882 raft_consensus.cc:1261] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 1 FOLLOWER]: Refusing update from remote peer c6ca92cd60d9414cbf8c933e8de1065a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:53:03.379215 15750 raft_consensus.cc:1261] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 1 FOLLOWER]: Refusing update from remote peer c6ca92cd60d9414cbf8c933e8de1065a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:53:03.380129 15950 consensus_queue.cc:1035] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:53:03.380450 15954 consensus_queue.cc:1035] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:53:03.390942 15958 mvcc.cc:205] Tried to move back new op lower bound from 7332705007110422528 to 7332705006905110528. Current Snapshot: MvccSnapshot[applied={T|T < 7332705007110422528}]
I20260924 00:53:03.405958 15473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:53:03.406268 15473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:53:03.410176 15473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Sending DeleteTablet for 3 replicas of tablet 7fd0edf6d3864ac5a8ef8b7ad1453063
I20260924 00:53:03.411538 15596 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:38262
I20260924 00:53:03.412411 15967 tablet_replica.cc:330] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: stopping tablet replica
I20260924 00:53:03.412536 15730 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:54468
I20260924 00:53:03.412933 15967 raft_consensus.cc:2229] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:53:03.412963 15862 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:03 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:49958
I20260924 00:53:03.413861 15967 raft_consensus.cc:2258] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:03.413900 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 15512
I20260924 00:53:03.425921 15969 tablet_replica.cc:330] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: stopping tablet replica
I20260924 00:53:03.426589 15969 raft_consensus.cc:2229] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:53:03.427270 15969 raft_consensus.cc:2258] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:03.428059 15968 tablet_replica.cc:330] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: stopping tablet replica
I20260924 00:53:03.428383 15968 raft_consensus.cc:2229] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:53:03.428787 15968 raft_consensus.cc:2258] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:03.430562 15969 ts_tablet_manager.cc:1905] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:03.431912 15967 ts_tablet_manager.cc:1905] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:03.433459 15968 ts_tablet_manager.cc:1905] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:03.434029 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.414708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.0.222.1:35071)
0924 00:53:03.414884 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.414890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.414910 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:53:03.415080 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.415086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.415166 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.415283 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:03.417256 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.418455 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.420404 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.421053 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.421173 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.425037 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:03.425051 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:03.425068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:03.425117 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:03.429668 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.430626 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.430635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.430639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.430823 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.431301 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.431306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.431308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.431770 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:03.432121 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.433393 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.433644 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:53:03.436621 15969 ts_tablet_manager.cc:1918] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:03.436769 15969 log.cc:1196] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/7fd0edf6d3864ac5a8ef8b7ad1453063
I20260924 00:53:03.437286 15657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:03.437413 15969 ts_tablet_manager.cc:1939] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 87844157e1374003816059043822dc21: Deleting consensus metadata
I20260924 00:53:03.439549 15462 catalog_manager.cc:4841] TS 87844157e1374003816059043822dc21 (127.0.222.3:36981): tablet 7fd0edf6d3864ac5a8ef8b7ad1453063 (table test-table [id=04d6bda70f8845c497a017cdbf261858]) successfully deleted
I20260924 00:53:03.440744 15968 ts_tablet_manager.cc:1918] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:03.440897 15968 log.cc:1196] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/7fd0edf6d3864ac5a8ef8b7ad1453063
I20260924 00:53:03.441350 15968 ts_tablet_manager.cc:1939] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P 7870ec61274e42e9b3c685d708fe9045: Deleting consensus metadata
I20260924 00:53:03.442811 15462 catalog_manager.cc:4841] TS 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281): tablet 7fd0edf6d3864ac5a8ef8b7ad1453063 (table test-table [id=04d6bda70f8845c497a017cdbf261858]) successfully deleted
I20260924 00:53:03.533058 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 15512
I20260924 00:53:03.548909 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 15666
W20260924 00:53:03.548985 15461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:35071 (error 108)
I20260924 00:53:03.550933 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.550541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38280)
0924 00:53:03.550688 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.550793 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 00:53:03.551167 15461 catalog_manager.cc:4584] TS c6ca92cd60d9414cbf8c933e8de1065a (127.0.222.1:35071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fd0edf6d3864ac5a8ef8b7ad1453063: Network error: Client connection negotiation failed: client connection to 127.0.222.1:35071: connect: Connection refused (error 111)
I20260924 00:53:03.567965 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.549866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.0.222.2:36281)
0924 00:53:03.550057 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.550064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.550085 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:53:03.550275 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.550280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.550367 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.550455 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:03.552043 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.552790 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.554442 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.554927 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.555042 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.557923 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:03.557940 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:03.557956 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:03.558010 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:03.565070 (+ 7060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.565918 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.565926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.565929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.566079 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.566419 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.566423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.566425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.566879 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:03.567104 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.567389 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.567571 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:53:03.568626 15790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:03.604918 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.604449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38282)
0924 00:53:03.604651 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.604745 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 00:53:03.605934 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.605600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38296)
0924 00:53:03.605723 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.605803 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 00:53:03.659334 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 15666
I20260924 00:53:03.666543 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.665998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38304)
0924 00:53:03.666246 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.666358 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 00:53:03.667675 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.667298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38318)
0924 00:53:03.667453 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.667540 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 00:53:03.675282 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 15798
I20260924 00:53:03.689534 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.675983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.0.222.3:36981)
0924 00:53:03.676158 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:03.676162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:03.676180 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:53:03.676293 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:03.676297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:03.676374 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:03.676456 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:03.678145 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.678987 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.680459 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:03.680857 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:03.680954 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:03.683860 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:03.683876 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:03.683893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:03.683929 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:03.686800 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.687547 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.687556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.687558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.687720 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:03.688111 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:03.688114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:03.688116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:03.688522 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:03.688742 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:03.688960 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 00:53:03.689132 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:53:03.690351 15922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:03.748322 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.747839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38328)
0924 00:53:03.748060 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.748148 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 00:53:03.749397 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:03.748961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:35071 (local address 127.0.0.1:38330)
0924 00:53:03.749150 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:03.749279 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:35071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 00:53:03.777946 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 15798
I20260924 00:53:03.794042 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 15442
I20260924 00:53:03.795205 15505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:03.905974 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 15442
I20260924 00:53:03.933607 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42639
--webserver_interface=127.0.222.62
--webserver_port=33927
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:42639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:04.265455 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:04.265733 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:04.265836 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:04.273007 15975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:53:04.273118 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:04.273190 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:04.273286 15975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:53:04.273339 15975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:53:04.283401 15975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:42639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:42639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=33927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:04.284793 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:04.286211 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:04.297977 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:04.297956 15982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:04.297926 15984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:04.299300 15975 server_base.cc:1034] running on GCE node
I20260924 00:53:04.300289 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:04.301919 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:04.303157 15975 hybrid_clock.cc:648] HybridClock initialized: now 1790211184303090 us; error 49 us; skew 500 ppm
I20260924 00:53:04.303519 15975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:04 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:04.306725 15975 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:53:04.308261 15975 webserver.cc:460] Webserver started at http://127.0.222.62:33927/ using document root <none> and password file <none>
I20260924 00:53:04.308943 15975 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:04.309059 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:04.314743 15975 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 00:53:04.317902 15991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:04.319373 15975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:53:04.319550 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "04ce98d104fb4061b530edbae7654c03"
format_stamp: "Formatted at 2026-09-24 00:52:58 on dist-test-slave-rtvz"
server_key: "283d35ab61044792c2498271c6283de7"
server_key_iv: "bb7517ea2a82cd6239379bf791b161c2"
server_key_version: "encryptionkey@0"
I20260924 00:53:04.320533 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:04.338961 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:04.343327 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:04.343591 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:04.360895 15975 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:42639
I20260924 00:53:04.360918 16042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:42639 every 8 connection(s)
I20260924 00:53:04.362526 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:53:04.369673 16043 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:53:04.372771 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 15975
I20260924 00:53:04.372951 16043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Bootstrap starting.
I20260924 00:53:04.373663 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:35071
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:42639
--webserver_port=37897
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:53:04.397794 16043 log.cc:824] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:04.418859 16043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:53:04.419553 16043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03: Bootstrap complete.
I20260924 00:53:04.433028 16043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:53:04.434392 16043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ce98d104fb4061b530edbae7654c03, State: Initialized, Role: FOLLOWER
I20260924 00:53:04.435190 16043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:53:04.435482 16043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:53:04.435604 16043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:53:04.435778 16043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:53:04.442054 16043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:53:04.442849 16043 leader_election.cc:302] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04ce98d104fb4061b530edbae7654c03; no voters:
I20260924 00:53:04.443578 16043 leader_election.cc:288] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 00:53:04.443740 16047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:53:04.444777 16047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [term 3 LEADER]: Becoming Leader. State: Replica: 04ce98d104fb4061b530edbae7654c03, State: Running, Role: LEADER
I20260924 00:53:04.445391 16047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } }
I20260924 00:53:04.445858 16043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:53:04.447780 16048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "04ce98d104fb4061b530edbae7654c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } } }
I20260924 00:53:04.448143 16048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: This master's current role is: LEADER
I20260924 00:53:04.451954 16049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04ce98d104fb4061b530edbae7654c03. Latest consensus state: current_term: 3 leader_uuid: "04ce98d104fb4061b530edbae7654c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ce98d104fb4061b530edbae7654c03" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 42639 } } }
I20260924 00:53:04.452272 16049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ce98d104fb4061b530edbae7654c03 [sys.catalog]: This master's current role is: LEADER
I20260924 00:53:04.453761 16052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:53:04.462489 16052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:53:04.464151 16052 catalog_manager.cc:1259] Loaded cluster ID: 4f797f09e0094f91a4e7dc4864cc0467
I20260924 00:53:04.464272 16052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 00:53:04.466240 16052 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0f4c7879ad google::LogMessage::Fail() at ??:0
@ 0x7f0f4c78a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0f4c7874de google::LogMessage::Flush() at ??:0
@ 0x7f0f4c788a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0f621ba512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0f621b7976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0f6223f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0f6223f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0f4f3d3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0f4f3d691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0f4f3d6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0f4f3b1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0f5268b6db start_thread at ??:0
@ 0x7f0f4b62161f clone at ??:0
W20260924 00:53:04.777417 16045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:04.777714 16045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:04.777882 16045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:04.785475 16045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:04.785598 16045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:04.785717 16045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:53:04.795746 16045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:35071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=37897
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:04.797042 16045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:04.798480 16045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:04.811750 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:04.811726 16085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:04.811786 16087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:04.812572 16045 server_base.cc:1034] running on GCE node
I20260924 00:53:04.813162 16045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:04.814159 16045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:04.815354 16045 hybrid_clock.cc:648] HybridClock initialized: now 1790211184815302 us; error 41 us; skew 500 ppm
I20260924 00:53:04.815672 16045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:04 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:04.818852 16045 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:53:04.820472 16045 webserver.cc:460] Webserver started at http://127.0.222.1:37897/ using document root <none> and password file <none>
I20260924 00:53:04.821486 16045 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:04.821640 16045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:04.829609 16045 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 00:53:04.834646 16094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:04.836280 16045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:53:04.836455 16045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "c6ca92cd60d9414cbf8c933e8de1065a"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "4716b35ec88ff7a67d072d7922c81ec7"
server_key_iv: "505c042b520c70ba649313711594cef0"
server_key_version: "encryptionkey@0"
I20260924 00:53:04.837522 16045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:04.869019 16045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:04.873329 16045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:04.873582 16045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:04.874541 16045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:04.876384 16101 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 00:53:04.882685 16045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:53:04.882911 16045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260924 00:53:04.883035 16045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:53:04.886370 16045 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:53:04.886453 16045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260924 00:53:04.886821 16101 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Bootstrap starting.
I20260924 00:53:04.895651 16101 log.cc:824] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:04.912351 16101 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:53:04.913254 16101 tablet_bootstrap.cc:492] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Bootstrap complete.
I20260924 00:53:04.913780 16101 ts_tablet_manager.cc:1397] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.008s
I20260924 00:53:04.922331 16045 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:35071
I20260924 00:53:04.922377 16209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:35071 every 8 connection(s)
I20260924 00:53:04.923672 16045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:53:04.923378 16101 raft_consensus.cc:348] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:04.924458 16101 raft_consensus.cc:729] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ca92cd60d9414cbf8c933e8de1065a, State: Initialized, Role: FOLLOWER
I20260924 00:53:04.925180 16101 consensus_queue.cc:260] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:04.926788 16101 ts_tablet_manager.cc:1428] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 00:53:04.929191 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:04.926822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:35911)
0924 00:53:04.927877 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:04.928170 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":449,"threads_started":1}
W20260924 00:53:04.929909 16210 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42639 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:42639: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
I20260924 00:53:04.930868 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:04.930448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:53459)
0924 00:53:04.930616 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:04.930752 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 00:53:04.931814 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:04.931441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:59531)
0924 00:53:04.931591 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:04.931665 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 00:53:04.932080 16210 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:53:04.932497 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 16045
I20260924 00:53:04.933296 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:36281
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:42639
--webserver_port=33975
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:05.257879 16214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:05.258234 16214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:05.258363 16214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:05.265831 16214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:05.265959 16214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:05.266072 16214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:53:05.276108 16214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:36281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=33975
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:05.277542 16214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:05.278874 16214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:05.292081 16219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:05.292040 16220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:05.291978 16222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:05.293008 16214 server_base.cc:1034] running on GCE node
I20260924 00:53:05.293678 16214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:05.294641 16214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:05.295859 16214 hybrid_clock.cc:648] HybridClock initialized: now 1790211185295805 us; error 47 us; skew 500 ppm
I20260924 00:53:05.296180 16214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:05 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:05.299355 16214 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:53:05.300933 16214 webserver.cc:460] Webserver started at http://127.0.222.2:33975/ using document root <none> and password file <none>
I20260924 00:53:05.301684 16214 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:05.301806 16214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:05.307312 16214 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 00:53:05.310289 16229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:05.311923 16214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:53:05.312125 16214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "7870ec61274e42e9b3c685d708fe9045"
format_stamp: "Formatted at 2026-09-24 00:52:59 on dist-test-slave-rtvz"
server_key: "598817166d4143008ebe1b11aeaf3ae5"
server_key_iv: "1f1d589ca979439a3bb898555763b54c"
server_key_version: "encryptionkey@0"
I20260924 00:53:05.313162 16214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:05.335626 16214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:05.339797 16214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:05.340068 16214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:05.341085 16214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:05.342552 16214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:05.342648 16214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:05.342752 16214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:05.342821 16214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:05.374002 16214 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:36281
I20260924 00:53:05.374069 16346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:36281 every 8 connection(s)
I20260924 00:53:05.375391 16214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:53:05.375998 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 16214
I20260924 00:53:05.376765 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:36981
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:42639
--webserver_port=43049
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:53:05.380887 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:05.378517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:54079)
0924 00:53:05.379374 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:05.379652 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":319,"threads_started":1}
W20260924 00:53:05.381635 16347 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42639 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:42639: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
I20260924 00:53:05.382611 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:05.382245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:45345)
0924 00:53:05.382422 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:05.382496 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 00:53:05.383431 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:05.383113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:34607)
0924 00:53:05.383250 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:05.383322 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 00:53:05.383643 16347 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:53:05.710819 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:05.711115 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:05.711210 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:05.718694 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:05.718822 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:05.718951 16350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:53:05.729063 16350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:36981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=43049
--tserver_master_addrs=127.0.222.62:42639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:05.730346 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:05.731640 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:05.745420 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:05.745393 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:05.745369 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:05.745923 16350 server_base.cc:1034] running on GCE node
I20260924 00:53:05.746695 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:05.747675 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:05.748903 16350 hybrid_clock.cc:648] HybridClock initialized: now 1790211185748816 us; error 52 us; skew 500 ppm
I20260924 00:53:05.749243 16350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:05 dist-test-slave-rtvz krb5kdc[14886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:05.752372 16350 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:53:05.753998 16350 webserver.cc:460] Webserver started at http://127.0.222.3:43049/ using document root <none> and password file <none>
I20260924 00:53:05.754674 16350 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:05.754791 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:05.760365 16350 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 00:53:05.763234 16365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:05.764642 16350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:53:05.764793 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "87844157e1374003816059043822dc21"
format_stamp: "Formatted at 2026-09-24 00:53:00 on dist-test-slave-rtvz"
server_key: "d60341c58f15ab4834f786ad6cee5142"
server_key_iv: "68eb2c54385b1aec6e5cce86b2fcae6d"
server_key_version: "encryptionkey@0"
I20260924 00:53:05.765854 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:05.784097 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:05.788997 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:05.789311 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:05.790323 16350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:05.791747 16350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:05.791836 16350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:05.791914 16350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:05.791972 16350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:05.822144 16350 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:36981
I20260924 00:53:05.822209 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:36981 every 8 connection(s)
I20260924 00:53:05.824071 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:53:05.830488 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:05.827846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:42003)
0924 00:53:05.828841 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:05.829122 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":357,"threads_started":1}
I20260924 00:53:05.830778 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 16350
W20260924 00:53:05.831902 16480 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:42639 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:42639: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
I20260924 00:53:05.832998 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:05.832535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:38005)
0924 00:53:05.832735 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:05.832832 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 00:53:05.834149 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:05.833628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:58433)
0924 00:53:05.833891 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:05.833993 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260924 00:53:05.834499 16480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:53:05.933439 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:05.932637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:35537)
0924 00:53:05.933069 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:05.933160 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":196,"threads_started":1}
I20260924 00:53:06.385219 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:06.384355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:47501)
0924 00:53:06.384826 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:06.384957 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":214,"threads_started":1}
I20260924 00:53:06.603715 16486 raft_consensus.cc:482] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:53:06.604012 16486 raft_consensus.cc:504] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:06.605572 16486 leader_election.cc:288] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281), 87844157e1374003816059043822dc21 (127.0.222.3:36981)
Sep 24 00:53:06 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
Sep 24 00:53:06 dist-test-slave-rtvz krb5kdc[14886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:53:06.628058 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:06.606136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:51871 (local address 127.0.222.2:36281)
0924 00:53:06.608244 (+ 2108us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:06.608261 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:06.608308 (+ 47us) server_negotiation.cc:407] Connection header received
0924 00:53:06.610583 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:06.610594 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:06.611122 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:06.612233 (+ 1111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:06.613574 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:06.614826 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:06.616570 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:06.617337 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:06.617505 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:06.621462 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:06.621472 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:06.621486 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:06.621528 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:06.624706 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:06.625449 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:06.625455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:06.625461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:06.625624 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:06.625993 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:06.625996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:06.625998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:06.626466 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:06.626633 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:06.627383 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 00:53:06.627769 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1893,"thread_start_us":1735,"threads_started":1}
I20260924 00:53:06.628111 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:06.606576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:36981 (local address 127.0.222.1:36911)
0924 00:53:06.607845 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:06.607863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:06.607974 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:06.611804 (+ 3830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:06.611809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:06.612678 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:06.613360 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:06.613384 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:06.614972 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:06.614975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:06.616414 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:06.616427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:06.617585 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:06.617592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:06.617808 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:06.618598 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:06.618627 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:06.620951 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:06.624749 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:06.624759 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:06.624774 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:06.625233 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:06.625652 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:06.625657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:06.625661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:06.625843 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:06.626598 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:06.626605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:06.626901 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 00:53:06.627135 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 00:53:06.627669 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":102,"spinlock_wait_cycles":1792,"thread_start_us":983,"threads_started":1}
I20260924 00:53:06.628135 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:06.606899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:36911 (local address 127.0.222.3:36981)
0924 00:53:06.608444 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:06.608458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:06.608489 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:53:06.610579 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:06.610594 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:06.611122 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:06.611522 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:06.613573 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:06.614759 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:06.616556 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:06.617333 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:06.617504 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:06.621292 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:06.621304 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:06.621324 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:06.621371 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:06.624613 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:06.625353 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:06.625363 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:06.625371 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:06.625540 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:06.625951 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:06.625956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:06.625959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:06.626467 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:06.626625 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:06.627357 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 00:53:06.627754 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"thread_start_us":1244,"threads_started":1}
I20260924 00:53:06.628772 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:06.605921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:36281 (local address 127.0.222.1:51871)
0924 00:53:06.606434 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:06.606481 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:06.607082 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:06.611714 (+ 4632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:06.611768 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:06.612723 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:06.613365 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:06.613385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:06.614996 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:06.615000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:06.616414 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:06.616427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:06.617566 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:06.617573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:06.617817 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:06.618778 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:06.618796 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:06.621190 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:06.624848 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:06.624853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:06.624858 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:06.625325 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:06.625736 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:06.625741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:06.625744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:06.625881 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:06.626599 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:06.626606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:06.626901 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 00:53:06.627133 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 00:53:06.628422 (+ 1289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":929,"spinlock_wait_cycles":12928,"thread_start_us":236,"threads_started":1}
W20260924 00:53:06.631569 16098 leader_election.cc:341] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87844157e1374003816059043822dc21 (127.0.222.3:36981): Not found: Tablet not found: 7fd0edf6d3864ac5a8ef8b7ad1453063
W20260924 00:53:06.631871 16098 leader_election.cc:341] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281): Not found: Tablet not found: 7fd0edf6d3864ac5a8ef8b7ad1453063
I20260924 00:53:06.631979 16098 leader_election.cc:302] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ca92cd60d9414cbf8c933e8de1065a; no voters: 7870ec61274e42e9b3c685d708fe9045, 87844157e1374003816059043822dc21
I20260924 00:53:06.632351 16486 raft_consensus.cc:2726] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:53:06.835968 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:06.835070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:52805)
0924 00:53:06.835576 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:06.835712 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":231,"threads_started":1}
I20260924 00:53:06.934968 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:06.934503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:48741)
0924 00:53:06.934682 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:06.934789 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 00:53:07.386919 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:07.386211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:45145)
0924 00:53:07.386660 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:07.386745 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":244,"threads_started":1}
I20260924 00:53:07.837838 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:07.836920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:57375)
0924 00:53:07.837566 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:07.837658 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":286,"threads_started":1}
I20260924 00:53:07.936542 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:07.935827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:44297)
0924 00:53:07.936262 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:07.936349 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260924 00:53:08.388610 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:08.387769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:58067)
0924 00:53:08.388312 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:08.388413 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":269,"threads_started":1}
I20260924 00:53:08.431233 16497 raft_consensus.cc:482] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:53:08.431496 16497 raft_consensus.cc:504] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:08.432255 16497 leader_election.cc:288] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281), 87844157e1374003816059043822dc21 (127.0.222.3:36981)
W20260924 00:53:08.433354 16098 leader_election.cc:341] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281): Not found: Tablet not found: 7fd0edf6d3864ac5a8ef8b7ad1453063
W20260924 00:53:08.433578 16098 leader_election.cc:341] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87844157e1374003816059043822dc21 (127.0.222.3:36981): Not found: Tablet not found: 7fd0edf6d3864ac5a8ef8b7ad1453063
I20260924 00:53:08.433674 16098 leader_election.cc:302] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ca92cd60d9414cbf8c933e8de1065a; no voters: 7870ec61274e42e9b3c685d708fe9045, 87844157e1374003816059043822dc21
I20260924 00:53:08.433960 16497 raft_consensus.cc:2726] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:53:08.839586 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:08.838738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:42619)
0924 00:53:08.839303 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:08.839403 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":271,"threads_started":1}
I20260924 00:53:08.938376 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:08.937619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:33357)
0924 00:53:08.938111 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:08.938202 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260924 00:53:09.390514 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:09.389782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:34605)
0924 00:53:09.390247 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:09.390333 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
I20260924 00:53:09.841455 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:09.840638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:60577)
0924 00:53:09.841118 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:09.841244 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":245,"threads_started":1}
I20260924 00:53:09.940502 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:09.939546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:48955)
0924 00:53:09.940148 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:09.940236 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":273,"threads_started":1}
I20260924 00:53:10.392480 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:10.391661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:43289)
0924 00:53:10.392203 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:10.392303 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":292,"threads_started":1}
I20260924 00:53:10.753296 16504 raft_consensus.cc:482] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:53:10.753544 16504 raft_consensus.cc:504] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:10.754225 16504 leader_election.cc:288] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281), 87844157e1374003816059043822dc21 (127.0.222.3:36981)
W20260924 00:53:10.755273 16098 leader_election.cc:341] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87844157e1374003816059043822dc21 (127.0.222.3:36981): Not found: Tablet not found: 7fd0edf6d3864ac5a8ef8b7ad1453063
W20260924 00:53:10.755496 16098 leader_election.cc:341] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281): Not found: Tablet not found: 7fd0edf6d3864ac5a8ef8b7ad1453063
I20260924 00:53:10.755610 16098 leader_election.cc:302] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ca92cd60d9414cbf8c933e8de1065a; no voters: 7870ec61274e42e9b3c685d708fe9045, 87844157e1374003816059043822dc21
I20260924 00:53:10.755859 16504 raft_consensus.cc:2726] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:53:10.843307 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:10.842545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:50927)
0924 00:53:10.843030 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:10.843118 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260924 00:53:10.942354 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:10.941519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:48125)
0924 00:53:10.942014 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:10.942126 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":219,"threads_started":1}
I20260924 00:53:11.394394 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:11.393608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:38273)
0924 00:53:11.394125 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:11.394208 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260924 00:53:11.845028 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:11.844182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:51985)
0924 00:53:11.844693 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:11.844837 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":266,"threads_started":1}
I20260924 00:53:11.944070 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:11.943276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:44381)
0924 00:53:11.943757 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:11.943847 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260924 00:53:12.396163 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:12.395312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:50355)
0924 00:53:12.395871 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:12.395965 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":239,"threads_started":1}
I20260924 00:53:12.846843 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:12.846043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:41419)
0924 00:53:12.846549 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:12.846649 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":216,"threads_started":1}
I20260924 00:53:12.946007 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:12.945153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:56545)
0924 00:53:12.945701 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:12.945824 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":230,"threads_started":1}
I20260924 00:53:13.398141 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:13.397393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:50571)
0924 00:53:13.397857 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:13.397957 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":224,"threads_started":1}
I20260924 00:53:13.848518 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:13.847734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:56987)
0924 00:53:13.848242 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:13.848327 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
I20260924 00:53:13.947721 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:13.946938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:34041)
0924 00:53:13.947415 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:13.947500 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260924 00:53:14.400046 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:14.399202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:50773)
0924 00:53:14.399760 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:14.399871 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":290,"threads_started":1}
I20260924 00:53:14.850335 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:14.849573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:53273)
0924 00:53:14.850084 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:14.850173 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":249,"threads_started":1}
I20260924 00:53:14.949786 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:14.948893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:50871)
0924 00:53:14.949494 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:14.949604 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":298,"threads_started":1}
I20260924 00:53:15.304828 16519 raft_consensus.cc:482] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:53:15.305056 16519 raft_consensus.cc:504] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7870ec61274e42e9b3c685d708fe9045" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 36281 } } peers { permanent_uuid: "87844157e1374003816059043822dc21" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 36981 } } peers { permanent_uuid: "c6ca92cd60d9414cbf8c933e8de1065a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 35071 } }
I20260924 00:53:15.305779 16519 leader_election.cc:288] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281), 87844157e1374003816059043822dc21 (127.0.222.3:36981)
W20260924 00:53:15.306646 16098 leader_election.cc:341] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7870ec61274e42e9b3c685d708fe9045 (127.0.222.2:36281): Not found: Tablet not found: 7fd0edf6d3864ac5a8ef8b7ad1453063
W20260924 00:53:15.306869 16098 leader_election.cc:341] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87844157e1374003816059043822dc21 (127.0.222.3:36981): Not found: Tablet not found: 7fd0edf6d3864ac5a8ef8b7ad1453063
I20260924 00:53:15.306952 16098 leader_election.cc:302] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ca92cd60d9414cbf8c933e8de1065a; no voters: 7870ec61274e42e9b3c685d708fe9045, 87844157e1374003816059043822dc21
I20260924 00:53:15.307204 16519 raft_consensus.cc:2726] T 7fd0edf6d3864ac5a8ef8b7ad1453063 P c6ca92cd60d9414cbf8c933e8de1065a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:53:15.401852 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:15.401104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:55973)
0924 00:53:15.401593 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:15.401677 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":275,"threads_started":1}
I20260924 00:53:15.852018 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:15.851275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:44625)
0924 00:53:15.851734 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:15.851823 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":229,"threads_started":1}
I20260924 00:53:15.951596 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:15.950818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:47401)
0924 00:53:15.951317 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:15.951412 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":255,"threads_started":1}
I20260924 00:53:16.403764 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:16.403005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:39563)
0924 00:53:16.403499 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:16.403590 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":225,"threads_started":1}
I20260924 00:53:16.853677 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:16.852858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:50795)
0924 00:53:16.853408 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:16.853512 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":224,"threads_started":1}
I20260924 00:53:16.953617 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:16.952635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:55963)
0924 00:53:16.953281 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:16.953399 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":301,"threads_started":1}
I20260924 00:53:17.405579 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:17.404737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:56775)
0924 00:53:17.405291 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:17.405374 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":221,"threads_started":1}
I20260924 00:53:17.855331 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:17.854599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:37335)
0924 00:53:17.855076 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:17.855160 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":224,"threads_started":1}
I20260924 00:53:17.955380 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:17.954622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:50791)
0924 00:53:17.955097 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:17.955188 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":237,"threads_started":1}
I20260924 00:53:18.407349 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:18.406514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:33263)
0924 00:53:18.407046 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:18.407180 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":286,"threads_started":1}
I20260924 00:53:18.857025 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:18.856287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:46091)
0924 00:53:18.856756 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:18.856859 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":228,"threads_started":1}
I20260924 00:53:18.957235 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:18.956451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:46747)
0924 00:53:18.956930 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:18.957023 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":255,"threads_started":1}
I20260924 00:53:19.409137 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:19.408315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:39631)
0924 00:53:19.408843 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:19.408940 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
I20260924 00:53:19.858875 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:19.858032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:39535)
0924 00:53:19.858505 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:19.858588 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":244,"threads_started":1}
I20260924 00:53:19.959547 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:19.958571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.1:45915)
0924 00:53:19.959137 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:19.959256 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":320,"threads_started":1}
I20260924 00:53:20.411141 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:20.410336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.2:35511)
0924 00:53:20.410864 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:20.410968 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260924 00:53:20.842535 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 16045
I20260924 00:53:20.861500 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:20.860023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:42639 (local address 127.0.222.3:48389)
0924 00:53:20.861054 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:20.861262 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:42639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":758,"threads_started":1}
I20260924 00:53:20.863108 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:20.843561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.0.222.1:35071)
0924 00:53:20.844134 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:20.844144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:20.844179 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:53:20.844260 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:20.844270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:20.844539 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:20.844674 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:20.846339 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:20.847403 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:20.848862 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:20.849513 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:20.849616 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:20.856637 (+ 7021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:20.856653 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:20.856670 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:20.856718 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:20.859829 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:20.860675 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:20.860687 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:20.860691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:20.860893 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:20.861454 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:20.861458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:20.861460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:20.861992 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:20.862245 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:20.862590 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 00:53:20.862864 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260924 00:53:20.864445 16205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:20.956040 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 16045
I20260924 00:53:20.972043 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 16214
I20260924 00:53:20.990376 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:20.972863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.222.2:36281)
0924 00:53:20.973340 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:20.973347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:20.975135 (+ 1788us) server_negotiation.cc:407] Connection header received
0924 00:53:20.975304 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:20.975309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:20.975383 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:20.975507 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:20.977161 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:20.978323 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:20.979590 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:20.980187 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:20.980324 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:20.983624 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:20.983629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:20.983633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:20.983671 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:20.987236 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:20.988094 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:20.988100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:20.988104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:20.988290 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:20.988725 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:20.988731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:20.988734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:20.989127 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:20.989429 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:20.989766 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 00:53:20.989991 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":223,"threads_started":1}
I20260924 00:53:20.991245 16341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:21.085012 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 16214
I20260924 00:53:21.100077 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 16350
I20260924 00:53:21.113740 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:21.100862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.0.222.3:36981)
0924 00:53:21.101428 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:21.101433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:21.101450 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:53:21.101507 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:21.101510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:21.101577 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:21.101663 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:21.103055 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:21.103832 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:21.105099 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:21.105685 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:21.105788 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:21.108445 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:21.108450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:21.108453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:21.108487 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:21.111081 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:21.111837 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:21.111843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:21.111845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:21.112051 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:21.112415 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:21.112419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:21.112421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:21.112694 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:21.112971 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:21.113244 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 00:53:21.113438 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":270,"threads_started":1}
I20260924 00:53:21.114467 16475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:21.200186 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 16350
2026-09-24T00:53:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24251 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:53:21 dist-test-slave-rtvz krb5kdc[16545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:53:21 dist-test-slave-rtvz krb5kdc[16545](info): set up 2 sockets
Sep 24 00:53:21 dist-test-slave-rtvz krb5kdc[16545](info): commencing operation
krb5kdc: starting...
W20260924 00:53:22.224889 888 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:53:22 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:53:22.461352 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33993
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:22.790867 16558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:22.791170 16558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:22.791292 16558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:22.799181 16558 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:53:22.799326 16558 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:53:22.799373 16558 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:53:22.799459 16558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:53:22.799547 16558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:22.799657 16558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:22.799738 16558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:53:22.799823 16558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:53:22.810377 16558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:22.811822 16558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:22.813566 16558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:22.825934 16566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:22.825984 16564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:22.826001 16563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:22.826977 16558 server_base.cc:1034] running on GCE node
I20260924 00:53:22.827932 16558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:22.829923 16558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:22.831204 16558 hybrid_clock.cc:648] HybridClock initialized: now 1790211202831124 us; error 46 us; skew 500 ppm
I20260924 00:53:22.831578 16558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:22 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:22.834721 16558 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:53:22.836208 16558 webserver.cc:460] Webserver started at http://127.0.222.62:44755/ using document root <none> and password file <none>
I20260924 00:53:22.836915 16558 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:22.837013 16558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:22.837370 16558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:53:22.840355 16558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "c0b025243c3549959ef7293ab5d8a5b2"
format_stamp: "Formatted at 2026-09-24 00:53:22 on dist-test-slave-rtvz"
server_key: "2e885ed48d2329184193bcc8057bdc01"
server_key_iv: "140128d8351d0b354303ca5d6b7da27c"
server_key_version: "encryptionkey@0"
I20260924 00:53:22.841141 16558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "c0b025243c3549959ef7293ab5d8a5b2"
format_stamp: "Formatted at 2026-09-24 00:53:22 on dist-test-slave-rtvz"
server_key: "2e885ed48d2329184193bcc8057bdc01"
server_key_iv: "140128d8351d0b354303ca5d6b7da27c"
server_key_version: "encryptionkey@0"
I20260924 00:53:22.846760 16558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 00:53:22.850756 16573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:22.852126 16558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:53:22.852298 16558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "c0b025243c3549959ef7293ab5d8a5b2"
format_stamp: "Formatted at 2026-09-24 00:53:22 on dist-test-slave-rtvz"
server_key: "2e885ed48d2329184193bcc8057bdc01"
server_key_iv: "140128d8351d0b354303ca5d6b7da27c"
server_key_version: "encryptionkey@0"
I20260924 00:53:22.852519 16558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:22.867420 16558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:22.871783 16558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:22.872035 16558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:22.889338 16558 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33993
I20260924 00:53:22.889344 16624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33993 every 8 connection(s)
I20260924 00:53:22.890705 16558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:53:22.894290 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 16558
I20260924 00:53:22.894407 16625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:22.894500 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:53:22.895104 888 external_mini_cluster.cc:1428] Setting key 04a274fea70903326bb996e22f51f62b
I20260924 00:53:22.908326 16625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Bootstrap starting.
Sep 24 00:53:22 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:22.911684 16625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:22.912962 16625 log.cc:824] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:22.916873 16625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: No bootstrap required, opened a new log
I20260924 00:53:22.921080 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:22.896987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.0.222.62:33993)
0924 00:53:22.897849 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:22.897864 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:22.897914 (+ 50us) server_negotiation.cc:407] Connection header received
0924 00:53:22.900572 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:22.900603 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:22.901143 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:22.901643 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:22.902967 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:22.904977 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:22.906399 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:22.907491 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:22.907675 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:22.912192 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:22.912203 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:22.912223 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:22.912270 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:22.916085 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:22.916907 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:22.916917 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:22.916926 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:22.917146 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:22.917632 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:22.917638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:22.917642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:22.918378 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:22.918660 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:22.919074 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 00:53:22.919584 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":377,"threads_started":1}
I20260924 00:53:22.926203 16625 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:53:22.926589 16625 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:22.926666 16625 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b025243c3549959ef7293ab5d8a5b2, State: Initialized, Role: FOLLOWER
I20260924 00:53:22.927201 16625 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:53:22.927410 16625 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:53:22.927511 16625 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:53:22.927659 16625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:22.930210 16625 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:53:22.930716 16625 leader_election.cc:302] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0b025243c3549959ef7293ab5d8a5b2; no voters:
I20260924 00:53:22.931303 16625 leader_election.cc:288] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:53:22.931437 16630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:53:22.932120 16630 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 1 LEADER]: Becoming Leader. State: Replica: c0b025243c3549959ef7293ab5d8a5b2, State: Running, Role: LEADER
I20260924 00:53:22.932710 16630 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:53:22.933576 16625 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:53:22.935564 16632 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0b025243c3549959ef7293ab5d8a5b2. Latest consensus state: current_term: 1 leader_uuid: "c0b025243c3549959ef7293ab5d8a5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } } }
I20260924 00:53:22.935990 16632 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:53:22.936326 16631 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0b025243c3549959ef7293ab5d8a5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } } }
I20260924 00:53:22.937444 16631 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:53:22.940603 16640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:53:22.945417 16640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:53:22.956063 16640 catalog_manager.cc:1347] Generated new cluster ID: fa44cbca641b44a68353982f5d5ade5a
I20260924 00:53:22.956223 16640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:53:22.967208 16640 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:53:22.968595 16640 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:53:22.975502 16640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Generated new TSK 0
I20260924 00:53:22.976613 16640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:53:23.036705 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33993
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:23.365949 16652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:23.366245 16652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:23.366371 16652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:23.373812 16652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:23.373957 16652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:23.374117 16652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:53:23.384265 16652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:23.385666 16652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:23.387038 16652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:23.400058 16657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:23.400187 16658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:23.400151 16660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:23.401561 16652 server_base.cc:1034] running on GCE node
I20260924 00:53:23.402153 16652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:23.403057 16652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:23.404268 16652 hybrid_clock.cc:648] HybridClock initialized: now 1790211203404218 us; error 34 us; skew 500 ppm
I20260924 00:53:23.404557 16652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:23 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:23.407721 16652 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:53:23.409338 16652 webserver.cc:460] Webserver started at http://127.0.222.1:44137/ using document root <none> and password file <none>
I20260924 00:53:23.410035 16652 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:23.410137 16652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:23.410414 16652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:53:23.413482 16652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "a41cd836079747b4a36ca1cfe9a4ed91"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "c3108b854c372596661542d01cd31c6b"
server_key_iv: "82a439186744b5630e7095ecc2393a6a"
server_key_version: "encryptionkey@0"
I20260924 00:53:23.414224 16652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "a41cd836079747b4a36ca1cfe9a4ed91"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "c3108b854c372596661542d01cd31c6b"
server_key_iv: "82a439186744b5630e7095ecc2393a6a"
server_key_version: "encryptionkey@0"
I20260924 00:53:23.419545 16652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:53:23.423424 16667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:23.424819 16652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:53:23.425006 16652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "a41cd836079747b4a36ca1cfe9a4ed91"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "c3108b854c372596661542d01cd31c6b"
server_key_iv: "82a439186744b5630e7095ecc2393a6a"
server_key_version: "encryptionkey@0"
I20260924 00:53:23.425241 16652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:23.445159 16652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:23.449710 16652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:23.449988 16652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:23.450968 16652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:23.452327 16652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:23.452401 16652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:23.452526 16652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:23.452582 16652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:23.484200 16652 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:38671
I20260924 00:53:23.484241 16780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:38671 every 8 connection(s)
I20260924 00:53:23.485496 16652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:53:23.493265 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 16652
I20260924 00:53:23.493494 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:53:23.494181 888 external_mini_cluster.cc:1428] Setting key e93aa1af661d0fbc4c3f68fa36f93641
Sep 24 00:53:23 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:23.505503 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:23.487945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:56965 (local address 127.0.222.62:33993)
0924 00:53:23.488389 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:23.488394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:23.489405 (+ 1011us) server_negotiation.cc:407] Connection header received
0924 00:53:23.491250 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:23.491256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:23.491346 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:23.491437 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:23.493049 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:23.493931 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:23.495834 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:23.496231 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:23.496354 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:23.499837 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:23.499846 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:23.499851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:23.499891 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:23.502399 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:23.503159 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:23.503164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:23.503167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:23.503296 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:23.503704 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:23.503708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:23.503710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:23.503974 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:23.504104 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:23.504825 (+ 721us) server_negotiation.cc:299] Negotiation successful
0924 00:53:23.505074 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":216,"threads_started":1}
I20260924 00:53:23.506556 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:23.488526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:56965)
0924 00:53:23.489253 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:23.489298 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:23.490987 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:23.491686 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:23.491695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:23.492244 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:23.492856 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:23.492883 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:23.494080 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:23.494086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:23.495701 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:23.495713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:23.496449 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:23.496457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:23.496674 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:23.497313 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:23.497340 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:23.499593 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:23.502552 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:23.502562 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:23.502580 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:23.503038 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:23.503401 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:23.503406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:23.503410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:23.503599 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:23.504095 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:23.504103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:23.504454 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 00:53:23.504716 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 00:53:23.505334 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":273,"threads_started":1}
I20260924 00:53:23.508316 16781 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33993
I20260924 00:53:23.508659 16781 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:23.509573 16781 heartbeater.cc:511] Master 127.0.222.62:33993 requested a full tablet report, sending...
I20260924 00:53:23.511888 16590 ts_manager.cc:194] Registered new tserver with Master: a41cd836079747b4a36ca1cfe9a4ed91 (127.0.222.1:38671)
I20260924 00:53:23.513393 16590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:56965
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:53:23.545526 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33993
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:23.878597 16787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:23.878901 16787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:23.878995 16787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:23.886513 16787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:23.886627 16787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:23.886751 16787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:53:23.901360 16787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:23.902951 16787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:23.904561 16787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:23.917798 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:23.917728 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:23.917779 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:23.918928 16787 server_base.cc:1034] running on GCE node
I20260924 00:53:23.919562 16787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:23.920578 16787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:23.921788 16787 hybrid_clock.cc:648] HybridClock initialized: now 1790211203921742 us; error 51 us; skew 500 ppm
I20260924 00:53:23.922148 16787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:23 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:23.925320 16787 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:53:23.926769 16787 webserver.cc:460] Webserver started at http://127.0.222.2:34207/ using document root <none> and password file <none>
I20260924 00:53:23.927501 16787 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:23.927623 16787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:23.927932 16787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:53:23.931052 16787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "f69d15061b8c4677a1684de56698e7f0"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "e90a889f8beb23f8876f94cd10ae16d9"
server_key_iv: "eaa9da4f32288474108985bdb413c9b0"
server_key_version: "encryptionkey@0"
I20260924 00:53:23.931836 16787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "f69d15061b8c4677a1684de56698e7f0"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "e90a889f8beb23f8876f94cd10ae16d9"
server_key_iv: "eaa9da4f32288474108985bdb413c9b0"
server_key_version: "encryptionkey@0"
I20260924 00:53:23.936969 16787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:53:23.940692 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:23.942193 16787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:53:23.942340 16787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "f69d15061b8c4677a1684de56698e7f0"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "e90a889f8beb23f8876f94cd10ae16d9"
server_key_iv: "eaa9da4f32288474108985bdb413c9b0"
server_key_version: "encryptionkey@0"
I20260924 00:53:23.942528 16787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:23.953075 16787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:23.957577 16787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:23.957815 16787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:23.958775 16787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:23.960080 16787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:23.960151 16787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:23.960232 16787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:23.960273 16787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:23.990787 16787 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41031
I20260924 00:53:23.990828 16915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41031 every 8 connection(s)
I20260924 00:53:23.992045 16787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:53:23.998529 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 16787
I20260924 00:53:23.998713 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:53:23.999150 888 external_mini_cluster.cc:1428] Setting key c320a2b5a1c109d2ad45bee73a843cf3
Sep 24 00:53:24 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:24.013175 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:23.994528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:33305 (local address 127.0.222.62:33993)
0924 00:53:23.994727 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:23.994733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:23.995908 (+ 1175us) server_negotiation.cc:407] Connection header received
0924 00:53:23.997837 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:23.997842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:23.997917 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:23.998002 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:23.999650 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.000539 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.002219 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.002688 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.002816 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.006504 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:24.006511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:24.006515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:24.006556 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:24.010047 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:24.010840 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:24.010846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:24.010850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:24.010999 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:24.011425 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:24.011430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:24.011434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:24.011771 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:24.011908 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.012630 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.012858 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:53:24.014189 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:23.995053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:33305)
0924 00:53:23.995752 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:23.995795 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:23.997576 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:23.998208 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:23.998216 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:23.998849 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:23.999475 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:23.999496 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.000689 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.000693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.002075 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.002092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.002928 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.002936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.003161 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.003831 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:24.003854 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:24.006239 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:24.010203 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:24.010211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:24.010225 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:24.010704 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:24.011105 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:24.011108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:24.011110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:24.011309 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:24.011907 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:24.011913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:24.012268 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 00:53:24.012502 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 00:53:24.012941 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":279,"threads_started":1}
I20260924 00:53:24.015707 16916 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33993
I20260924 00:53:24.016062 16916 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:24.016917 16916 heartbeater.cc:511] Master 127.0.222.62:33993 requested a full tablet report, sending...
I20260924 00:53:24.018671 16590 ts_manager.cc:194] Registered new tserver with Master: f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031)
I20260924 00:53:24.019490 16590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:33305
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:53:24.052625 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33993
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:24.375172 16921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:24.375483 16921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:24.375579 16921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:24.382786 16921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:24.382910 16921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:24.383024 16921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:53:24.392834 16921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:24.394212 16921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:24.395468 16921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:24.408206 16926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:24.408170 16929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:24.408239 16927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:24.409050 16921 server_base.cc:1034] running on GCE node
I20260924 00:53:24.409662 16921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:24.410615 16921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:24.411824 16921 hybrid_clock.cc:648] HybridClock initialized: now 1790211204411767 us; error 44 us; skew 500 ppm
I20260924 00:53:24.412129 16921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:24 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:24.415062 16921 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:53:24.416540 16921 webserver.cc:460] Webserver started at http://127.0.222.3:43581/ using document root <none> and password file <none>
I20260924 00:53:24.417236 16921 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:24.417372 16921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:24.417685 16921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:53:24.420652 16921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "821c0109a8d84220a89985b2e6bc2d1e"
format_stamp: "Formatted at 2026-09-24 00:53:24 on dist-test-slave-rtvz"
server_key: "aeaf5af357b0697b2cc63063c5d91876"
server_key_iv: "b34bae93169bac9e9eceaf963864cd57"
server_key_version: "encryptionkey@0"
I20260924 00:53:24.421466 16921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "821c0109a8d84220a89985b2e6bc2d1e"
format_stamp: "Formatted at 2026-09-24 00:53:24 on dist-test-slave-rtvz"
server_key: "aeaf5af357b0697b2cc63063c5d91876"
server_key_iv: "b34bae93169bac9e9eceaf963864cd57"
server_key_version: "encryptionkey@0"
I20260924 00:53:24.426690 16921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 00:53:24.430475 16936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:24.431887 16921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:53:24.432053 16921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "821c0109a8d84220a89985b2e6bc2d1e"
format_stamp: "Formatted at 2026-09-24 00:53:24 on dist-test-slave-rtvz"
server_key: "aeaf5af357b0697b2cc63063c5d91876"
server_key_iv: "b34bae93169bac9e9eceaf963864cd57"
server_key_version: "encryptionkey@0"
I20260924 00:53:24.432246 16921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:24.454317 16921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:24.459046 16921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:24.459347 16921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:24.460376 16921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:24.461795 16921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:24.461884 16921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:24.462005 16921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:24.462059 16921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:24.492254 16921 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:44265
I20260924 00:53:24.492312 17048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:44265 every 8 connection(s)
I20260924 00:53:24.493561 16921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:53:24.495747 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 16921
I20260924 00:53:24.495972 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:53:24.496477 888 external_mini_cluster.cc:1428] Setting key 848570d97d9a435106ec1a49eff3325c
Sep 24 00:53:24 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:24.512993 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.496124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:58865 (local address 127.0.222.62:33993)
0924 00:53:24.496350 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:24.496357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:24.497611 (+ 1254us) server_negotiation.cc:407] Connection header received
0924 00:53:24.499740 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:24.499745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:24.499832 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:24.499910 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:24.501401 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.502132 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.503726 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.504101 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.504186 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.507490 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:24.507496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:24.507499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:24.507532 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:24.509985 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:24.510775 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:24.510779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:24.510781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:24.510905 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:24.511314 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:24.511317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:24.511320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:24.511606 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:24.511737 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.512444 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.512653 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:53:24.513970 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.496610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:58865)
0924 00:53:24.497457 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:24.497504 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:24.499470 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:24.500126 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:24.500134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:24.500632 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:24.501181 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.501236 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.502283 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.502288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.503617 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.503629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.504299 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.504305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.504511 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.505151 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:24.505173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:24.507294 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:24.510122 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:24.510131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:24.510149 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:24.510662 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:24.511007 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:24.511012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:24.511016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:24.511199 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:24.511726 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:24.511734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:24.512086 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 00:53:24.512312 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 00:53:24.512828 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":332,"threads_started":1}
I20260924 00:53:24.515264 17049 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33993
I20260924 00:53:24.515529 17049 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:24.516192 17049 heartbeater.cc:511] Master 127.0.222.62:33993 requested a full tablet report, sending...
I20260924 00:53:24.516961 16781 heartbeater.cc:503] Master 127.0.222.62:33993 was elected leader, sending a full tablet report...
I20260924 00:53:24.517668 16590 ts_manager.cc:194] Registered new tserver with Master: 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265)
I20260924 00:53:24.518663 16590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:58865
I20260924 00:53:24.523954 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:53:24.540278 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.528234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.222.62:33993)
0924 00:53:24.528458 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:24.528465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:24.528789 (+ 324us) server_negotiation.cc:407] Connection header received
0924 00:53:24.528970 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:24.528974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:24.529034 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:24.529128 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:24.530562 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.531259 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.532598 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.532973 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.533064 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.535564 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:24.535569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:24.535572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:24.535601 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:24.538042 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:24.538686 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:24.538690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:24.538692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:24.538811 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:24.539101 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:24.539104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:24.539106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:24.539416 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:24.539598 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.539866 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.540023 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 00:53:24.544121 16590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:53:24.551199 16590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:53:24.577732 17065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.563912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41031 (local address 127.0.0.1:52936)
0924 00:53:24.565349 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:24.565371 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:24.565480 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:24.570964 (+ 5484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:24.570971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:24.571027 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:24.571445 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.571455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.573187 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.573192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.575377 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.575391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.576713 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.576719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.576918 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.576960 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 00:53:24.577082 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 00:53:24.577287 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":75,"spinlock_wait_cycles":2432,"thread_start_us":6506,"threads_started":1}
I20260924 00:53:24.578565 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.564479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.0.222.2:41031)
0924 00:53:24.570190 (+ 5711us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:24.570201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:24.570240 (+ 39us) server_negotiation.cc:407] Connection header received
0924 00:53:24.570367 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:24.570377 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:24.570681 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:24.571644 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:24.571872 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.573036 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.575521 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.576513 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.576636 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.576850 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.578081 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.578268 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5517,"thread_start_us":5380,"threads_started":1}
I20260924 00:53:24.579360 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.563460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:44265 (local address 127.0.0.1:37610)
0924 00:53:24.564326 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:24.564381 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:24.564558 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:24.570962 (+ 6404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:24.570977 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:24.571014 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:24.571589 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.571602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.573897 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.573902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.576540 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.576556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.578482 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.578488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.578650 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.578680 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 00:53:24.578742 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:53:24.578880 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":88,"spinlock_wait_cycles":4352,"thread_start_us":453,"threads_started":1}
I20260924 00:53:24.579535 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.564769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:46218)
0924 00:53:24.570685 (+ 5916us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:24.570711 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:24.570874 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:24.571555 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:24.571559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:24.571587 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:24.571948 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.571966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.573900 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.573904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.576555 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.576565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.578715 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.578721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.578872 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.578914 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 00:53:24.579007 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 00:53:24.579132 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5788,"mutex_wait_us":48,"thread_start_us":789,"threads_started":1}
I20260924 00:53:24.579547 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.565379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.0.222.1:38671)
0924 00:53:24.569838 (+ 4459us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:24.569847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:24.571007 (+ 1160us) server_negotiation.cc:407] Connection header received
0924 00:53:24.571116 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:24.571123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:24.571348 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:24.571555 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:24.572135 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.573743 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.576755 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.578513 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.578632 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.578852 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.579062 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.579238 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4257,"thread_start_us":4060,"threads_started":1}
I20260924 00:53:24.580724 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.563793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.0.222.3:44265)
0924 00:53:24.570328 (+ 6535us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:24.570336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:24.570373 (+ 37us) server_negotiation.cc:407] Connection header received
0924 00:53:24.570473 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:24.570481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:24.570755 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:24.571796 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:24.572031 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.573740 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.576748 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.578267 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.579974 (+ 1707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.580188 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.580250 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.580415 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6361,"thread_start_us":6269,"threads_started":1}
I20260924 00:53:24.582757 16850 tablet_service.cc:1467] Processing CreateTablet for tablet c2bb8c5a9e7240479fa808a9ffc57f1c (DEFAULT_TABLE table=test-table [id=567199bf899c4f4aa528bbbeb61857cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:24.583184 16715 tablet_service.cc:1467] Processing CreateTablet for tablet c2bb8c5a9e7240479fa808a9ffc57f1c (DEFAULT_TABLE table=test-table [id=567199bf899c4f4aa528bbbeb61857cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:24.583760 16984 tablet_service.cc:1467] Processing CreateTablet for tablet c2bb8c5a9e7240479fa808a9ffc57f1c (DEFAULT_TABLE table=test-table [id=567199bf899c4f4aa528bbbeb61857cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:24.584267 16850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2bb8c5a9e7240479fa808a9ffc57f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:24.584511 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2bb8c5a9e7240479fa808a9ffc57f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:24.584843 16984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2bb8c5a9e7240479fa808a9ffc57f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:24.596012 17067 tablet_bootstrap.cc:492] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Bootstrap starting.
I20260924 00:53:24.598773 17068 tablet_bootstrap.cc:492] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Bootstrap starting.
I20260924 00:53:24.599903 17067 tablet_bootstrap.cc:654] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:24.601258 17067 log.cc:824] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:24.601291 17069 tablet_bootstrap.cc:492] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Bootstrap starting.
I20260924 00:53:24.602505 17068 tablet_bootstrap.cc:654] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:24.603890 17068 log.cc:824] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:24.604156 17069 tablet_bootstrap.cc:654] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:24.605309 17069 log.cc:824] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:24.606877 17068 tablet_bootstrap.cc:492] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: No bootstrap required, opened a new log
I20260924 00:53:24.607117 17068 ts_tablet_manager.cc:1397] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 00:53:24.607808 17067 tablet_bootstrap.cc:492] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: No bootstrap required, opened a new log
I20260924 00:53:24.608068 17069 tablet_bootstrap.cc:492] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: No bootstrap required, opened a new log
I20260924 00:53:24.608094 17067 ts_tablet_manager.cc:1397] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 00:53:24.608372 17069 ts_tablet_manager.cc:1397] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260924 00:53:24.614931 17068 raft_consensus.cc:348] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.615386 17068 raft_consensus.cc:374] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:24.615509 17068 raft_consensus.cc:729] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69d15061b8c4677a1684de56698e7f0, State: Initialized, Role: FOLLOWER
I20260924 00:53:24.616139 17068 consensus_queue.cc:260] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.617339 16916 heartbeater.cc:503] Master 127.0.222.62:33993 was elected leader, sending a full tablet report...
I20260924 00:53:24.618103 17068 ts_tablet_manager.cc:1428] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 00:53:24.618912 17069 raft_consensus.cc:348] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.619007 17067 raft_consensus.cc:348] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.619383 17067 raft_consensus.cc:374] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:24.619395 17069 raft_consensus.cc:374] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:24.619475 17067 raft_consensus.cc:729] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821c0109a8d84220a89985b2e6bc2d1e, State: Initialized, Role: FOLLOWER
I20260924 00:53:24.619475 17069 raft_consensus.cc:729] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41cd836079747b4a36ca1cfe9a4ed91, State: Initialized, Role: FOLLOWER
I20260924 00:53:24.620103 17069 consensus_queue.cc:260] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.620256 17067 consensus_queue.cc:260] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.622136 17069 ts_tablet_manager.cc:1428] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 00:53:24.623607 17049 heartbeater.cc:503] Master 127.0.222.62:33993 was elected leader, sending a full tablet report...
I20260924 00:53:24.624352 17067 ts_tablet_manager.cc:1428] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
W20260924 00:53:24.737860 16782 tablet.cc:2369] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:53:24.744052 16917 tablet.cc:2369] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:53:24.745108 17050 tablet.cc:2369] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:53:24.841089 17075 raft_consensus.cc:482] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:53:24.841449 17075 raft_consensus.cc:504] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.843109 17075 leader_election.cc:288] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41cd836079747b4a36ca1cfe9a4ed91 (127.0.222.1:38671), f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031)
I20260924 00:53:24.848762 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.843929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:43941 (local address 127.0.222.2:41031)
0924 00:53:24.844101 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:24.844105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:24.844121 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:53:24.844184 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:24.844188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:24.844243 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:24.844319 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:24.844727 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.845481 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.847097 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.847927 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.848057 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.848152 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.848389 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.848514 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 00:53:24.848783 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.843449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41031 (local address 127.0.222.3:43941)
0924 00:53:24.843649 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:24.843668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:24.843804 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:24.844375 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:24.844380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:24.844413 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:24.844593 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.844601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.845614 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.845618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.846943 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.846958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.848075 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.848080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.848249 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.848273 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 00:53:24.848359 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 00:53:24.848490 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"spinlock_wait_cycles":34688}
I20260924 00:53:24.849574 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.843585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:40757 (local address 127.0.222.1:38671)
0924 00:53:24.843792 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:24.843799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:24.844436 (+ 637us) server_negotiation.cc:407] Connection header received
0924 00:53:24.844600 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:24.844604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:24.844663 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:24.844735 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:24.845372 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.846134 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.847849 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.848643 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.848762 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.848858 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.849187 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.849343 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:53:24.849608 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.843566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.222.3:40757)
0924 00:53:24.844324 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:24.844339 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:24.844466 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:24.844845 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:24.844849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:24.844881 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:24.845182 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.845191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.846264 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.846269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.847693 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:24.847705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.848847 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.848852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:24.849039 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.849065 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 00:53:24.849142 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:53:24.849291 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":436,"threads_started":1}
I20260924 00:53:24.849818 16870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2bb8c5a9e7240479fa808a9ffc57f1c" candidate_uuid: "821c0109a8d84220a89985b2e6bc2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69d15061b8c4677a1684de56698e7f0" is_pre_election: true
I20260924 00:53:24.850414 16870 raft_consensus.cc:2454] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821c0109a8d84220a89985b2e6bc2d1e in term 0.
I20260924 00:53:24.850598 16735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2bb8c5a9e7240479fa808a9ffc57f1c" candidate_uuid: "821c0109a8d84220a89985b2e6bc2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" is_pre_election: true
I20260924 00:53:24.851123 16735 raft_consensus.cc:2454] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821c0109a8d84220a89985b2e6bc2d1e in term 0.
I20260924 00:53:24.851244 16938 leader_election.cc:302] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 821c0109a8d84220a89985b2e6bc2d1e, f69d15061b8c4677a1684de56698e7f0; no voters:
I20260924 00:53:24.851640 17075 raft_consensus.cc:2781] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:53:24.851763 17075 raft_consensus.cc:482] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:53:24.851862 17075 raft_consensus.cc:3037] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:24.854058 17075 raft_consensus.cc:504] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.854650 17075 leader_election.cc:288] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [CANDIDATE]: Term 1 election: Requested vote from peers a41cd836079747b4a36ca1cfe9a4ed91 (127.0.222.1:38671), f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031)
I20260924 00:53:24.855020 16735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2bb8c5a9e7240479fa808a9ffc57f1c" candidate_uuid: "821c0109a8d84220a89985b2e6bc2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41cd836079747b4a36ca1cfe9a4ed91"
I20260924 00:53:24.855129 16870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2bb8c5a9e7240479fa808a9ffc57f1c" candidate_uuid: "821c0109a8d84220a89985b2e6bc2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69d15061b8c4677a1684de56698e7f0"
I20260924 00:53:24.855228 16735 raft_consensus.cc:3037] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:24.855326 16870 raft_consensus.cc:3037] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:24.858374 16735 raft_consensus.cc:2454] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821c0109a8d84220a89985b2e6bc2d1e in term 1.
I20260924 00:53:24.858381 16870 raft_consensus.cc:2454] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821c0109a8d84220a89985b2e6bc2d1e in term 1.
I20260924 00:53:24.858788 16940 leader_election.cc:302] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 821c0109a8d84220a89985b2e6bc2d1e, a41cd836079747b4a36ca1cfe9a4ed91; no voters:
I20260924 00:53:24.859081 17075 raft_consensus.cc:2781] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:53:24.859493 17075 raft_consensus.cc:686] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 1 LEADER]: Becoming Leader. State: Replica: 821c0109a8d84220a89985b2e6bc2d1e, State: Running, Role: LEADER
I20260924 00:53:24.859958 17075 consensus_queue.cc:237] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } }
I20260924 00:53:24.864658 16589 catalog_manager.cc:5495] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3). New cstate: current_term: 1 leader_uuid: "821c0109a8d84220a89985b2e6bc2d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } health_report { overall_health: UNKNOWN } } }
I20260924 00:53:24.947638 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:24.942274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37616 (local address 127.0.222.3:44265)
0924 00:53:24.942477 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:24.942482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:24.942561 (+ 79us) server_negotiation.cc:407] Connection header received
0924 00:53:24.942717 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:24.942721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:24.942810 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:24.942961 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:53:24.943280 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.944014 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.945484 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:24.945910 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:24.946005 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:24.946495 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:53:24.947077 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:53:24.947186 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:24.947293 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 00:53:24.947369 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 00:53:24.956386 16735 raft_consensus.cc:1261] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Refusing update from remote peer 821c0109a8d84220a89985b2e6bc2d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:53:24.956386 16870 raft_consensus.cc:1261] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 1 FOLLOWER]: Refusing update from remote peer 821c0109a8d84220a89985b2e6bc2d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:53:24.957536 17077 consensus_queue.cc:1035] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:53:24.957911 17075 consensus_queue.cc:1035] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:53:24.969265 17081 mvcc.cc:205] Tried to move back new op lower bound from 7332705095490392064 to 7332705095109844992. Current Snapshot: MvccSnapshot[applied={T|T < 7332705095490392064}]
I20260924 00:53:24.988591 16589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:53:24.988925 16589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:53:24.993122 16589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Sending DeleteTablet for 3 replicas of tablet c2bb8c5a9e7240479fa808a9ffc57f1c
I20260924 00:53:24.994537 16984 tablet_service.cc:1514] Processing DeleteTablet for tablet c2bb8c5a9e7240479fa808a9ffc57f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:24 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:37610
I20260924 00:53:24.994789 16850 tablet_service.cc:1514] Processing DeleteTablet for tablet c2bb8c5a9e7240479fa808a9ffc57f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:24 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:52936
I20260924 00:53:24.995519 16715 tablet_service.cc:1514] Processing DeleteTablet for tablet c2bb8c5a9e7240479fa808a9ffc57f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:24 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:46218
I20260924 00:53:25.010012 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 16652
I20260924 00:53:25.016024 17093 tablet_replica.cc:330] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: stopping tablet replica
I20260924 00:53:25.016561 17093 raft_consensus.cc:2229] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:53:25.017694 17092 tablet_replica.cc:330] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: stopping tablet replica
I20260924 00:53:25.017988 17091 tablet_replica.cc:330] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: stopping tablet replica
I20260924 00:53:25.018121 17092 raft_consensus.cc:2229] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:53:25.018414 17091 raft_consensus.cc:2229] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:53:25.018626 17092 raft_consensus.cc:2258] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:25.018970 17093 raft_consensus.cc:2258] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:25.019274 17091 raft_consensus.cc:2258] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:25.021312 17091 ts_tablet_manager.cc:1905] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:25.022822 17092 ts_tablet_manager.cc:1905] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:25.025413 17093 ts_tablet_manager.cc:1905] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:25.029179 17091 ts_tablet_manager.cc:1918] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:25.029398 17091 log.cc:1196] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/c2bb8c5a9e7240479fa808a9ffc57f1c
I20260924 00:53:25.030122 17091 ts_tablet_manager.cc:1939] T c2bb8c5a9e7240479fa808a9ffc57f1c P 821c0109a8d84220a89985b2e6bc2d1e: Deleting consensus metadata
I20260924 00:53:25.032148 16577 catalog_manager.cc:4841] TS 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265): tablet c2bb8c5a9e7240479fa808a9ffc57f1c (table test-table [id=567199bf899c4f4aa528bbbeb61857cf]) successfully deleted
I20260924 00:53:25.033097 17093 ts_tablet_manager.cc:1918] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:25.033310 17093 log.cc:1196] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/c2bb8c5a9e7240479fa808a9ffc57f1c
Sep 24 00:53:25 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:53:25.033818 17093 ts_tablet_manager.cc:1939] T c2bb8c5a9e7240479fa808a9ffc57f1c P a41cd836079747b4a36ca1cfe9a4ed91: Deleting consensus metadata
I20260924 00:53:25.035705 16577 catalog_manager.cc:4841] TS a41cd836079747b4a36ca1cfe9a4ed91 (127.0.222.1:38671): tablet c2bb8c5a9e7240479fa808a9ffc57f1c (table test-table [id=567199bf899c4f4aa528bbbeb61857cf]) successfully deleted
I20260924 00:53:25.036373 17092 ts_tablet_manager.cc:1918] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:25.036621 17092 log.cc:1196] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/c2bb8c5a9e7240479fa808a9ffc57f1c
I20260924 00:53:25.037051 17092 ts_tablet_manager.cc:1939] T c2bb8c5a9e7240479fa808a9ffc57f1c P f69d15061b8c4677a1684de56698e7f0: Deleting consensus metadata
I20260924 00:53:25.038661 16575 catalog_manager.cc:4841] TS f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031): tablet c2bb8c5a9e7240479fa808a9ffc57f1c (table test-table [id=567199bf899c4f4aa528bbbeb61857cf]) successfully deleted
I20260924 00:53:25.042554 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:25.011790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.0.222.1:38671)
0924 00:53:25.011995 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:25.012001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:25.021391 (+ 9390us) server_negotiation.cc:407] Connection header received
0924 00:53:25.021955 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:25.021961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:25.022043 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:25.022117 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:25.024198 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:25.025344 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:25.027500 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:25.028163 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:25.028266 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:25.035325 (+ 7059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:25.035342 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:25.035362 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:25.035411 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:25.039233 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:25.040185 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:25.040195 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:25.040199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:25.040364 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:25.040747 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:25.040751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:25.040754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:25.041239 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:25.041588 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:25.041964 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 00:53:25.042180 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 00:53:25.043975 16776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:25.134207 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 16652
I20260924 00:53:25.150199 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 16787
Sep 24 00:53:25 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:53:25.167848 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:25.150984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.0.222.2:41031)
0924 00:53:25.151151 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:25.151158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:25.151220 (+ 62us) server_negotiation.cc:407] Connection header received
0924 00:53:25.151510 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:25.151516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:25.151626 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:25.151749 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:25.153503 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:25.154290 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:25.156357 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:25.156770 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:25.156885 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:25.161981 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:25.161998 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:25.162015 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:25.162056 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:25.164831 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:25.165682 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:25.165690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:25.165692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:25.165846 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:25.166249 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:25.166253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:25.166255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:25.166645 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:25.166926 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:25.167283 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 00:53:25.167527 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:53:25.168555 16910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:25.256599 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 16787
I20260924 00:53:25.271715 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 16921
Sep 24 00:53:25 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:53:25.287638 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:25.272468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.0.222.3:44265)
0924 00:53:25.272658 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:25.272664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:25.272744 (+ 80us) server_negotiation.cc:407] Connection header received
0924 00:53:25.272944 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:25.272950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:25.273028 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:25.273110 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:25.274431 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:25.275243 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:25.276650 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:25.277130 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:25.277265 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:25.281708 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:25.281723 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:25.281743 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:25.281779 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:25.284760 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:25.285523 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:25.285530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:25.285532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:25.285708 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:25.286084 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:25.286090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:25.286092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:25.286453 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:25.286751 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:25.287059 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 00:53:25.287274 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 00:53:25.288343 17044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:25.409562 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 16921
I20260924 00:53:25.426887 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 16558
I20260924 00:53:25.428045 16620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:25.535777 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 16558
I20260924 00:53:45.553617 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33993
--webserver_interface=127.0.222.62
--webserver_port=44755
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:53:45.895829 17099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:45.896174 17099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:45.896268 17099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:45.903749 17099 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:53:45.903864 17099 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:53:45.903904 17099 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:53:45.903950 17099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:53:45.904003 17099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:45.904069 17099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:45.904114 17099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:53:45.904165 17099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:53:45.914968 17099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=44755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:45.916326 17099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:45.917685 17099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:45.940454 17105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:45.940439 17106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:45.940603 17108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:45.941465 17099 server_base.cc:1034] running on GCE node
I20260924 00:53:45.942394 17099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:45.944054 17099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:45.945365 17099 hybrid_clock.cc:648] HybridClock initialized: now 1790211225945246 us; error 51 us; skew 500 ppm
I20260924 00:53:45.945773 17099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:45 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:45.949638 17099 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:53:45.951411 17099 webserver.cc:460] Webserver started at http://127.0.222.62:44755/ using document root <none> and password file <none>
I20260924 00:53:45.952262 17099 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:45.952399 17099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:45.959431 17099 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:53:45.963037 17115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:45.964787 17099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260924 00:53:45.965034 17099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "c0b025243c3549959ef7293ab5d8a5b2"
format_stamp: "Formatted at 2026-09-24 00:53:22 on dist-test-slave-rtvz"
server_key: "2e885ed48d2329184193bcc8057bdc01"
server_key_iv: "140128d8351d0b354303ca5d6b7da27c"
server_key_version: "encryptionkey@0"
I20260924 00:53:45.966318 17099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:45.988458 17099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:45.993593 17099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:45.993934 17099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:46.014411 17099 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33993
I20260924 00:53:46.014444 17167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33993 every 8 connection(s)
I20260924 00:53:46.016155 17099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:53:46.018385 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 17099
I20260924 00:53:46.019438 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:38671
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:33993
--webserver_port=44137
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:53:46.025859 17168 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:53:46.030833 17168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Bootstrap starting.
I20260924 00:53:46.052582 17168 log.cc:824] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:46.067232 17168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:53:46.067880 17168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Bootstrap complete.
I20260924 00:53:46.077927 17168 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:53:46.078884 17168 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b025243c3549959ef7293ab5d8a5b2, State: Initialized, Role: FOLLOWER
I20260924 00:53:46.079619 17168 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:53:46.079908 17168 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:53:46.080042 17168 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:53:46.080220 17168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:53:46.084493 17168 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:53:46.085101 17168 leader_election.cc:302] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0b025243c3549959ef7293ab5d8a5b2; no voters:
I20260924 00:53:46.085811 17168 leader_election.cc:288] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:53:46.085964 17172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:53:46.086701 17172 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 2 LEADER]: Becoming Leader. State: Replica: c0b025243c3549959ef7293ab5d8a5b2, State: Running, Role: LEADER
I20260924 00:53:46.087380 17172 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:53:46.088090 17168 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:53:46.090127 17173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0b025243c3549959ef7293ab5d8a5b2. Latest consensus state: current_term: 2 leader_uuid: "c0b025243c3549959ef7293ab5d8a5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } } }
I20260924 00:53:46.090626 17173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:53:46.093340 17174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c0b025243c3549959ef7293ab5d8a5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } } }
I20260924 00:53:46.093832 17174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:53:46.094911 17177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:53:46.103137 17177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:53:46.104619 17177 catalog_manager.cc:1259] Loaded cluster ID: fa44cbca641b44a68353982f5d5ade5a
I20260924 00:53:46.104791 17177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:53:46.109230 17177 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:53:46.111090 17177 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Loaded TSK: 0
I20260924 00:53:46.151628 17177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Generated new TSK 1
I20260924 00:53:46.152215 17177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:53:46.374861 17170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:46.375168 17170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:46.375303 17170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:46.382750 17170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:46.382882 17170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:46.383036 17170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:53:46.393129 17170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:38671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=44137
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:46.394513 17170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:46.395823 17170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:46.410202 17195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:46.410223 17196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:46.410899 17198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:46.411578 17170 server_base.cc:1034] running on GCE node
I20260924 00:53:46.412209 17170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:46.413250 17170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:46.414480 17170 hybrid_clock.cc:648] HybridClock initialized: now 1790211226414386 us; error 69 us; skew 500 ppm
I20260924 00:53:46.414796 17170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:46 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:46.421973 17170 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:53:46.423626 17170 webserver.cc:460] Webserver started at http://127.0.222.1:44137/ using document root <none> and password file <none>
I20260924 00:53:46.424322 17170 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:46.424461 17170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:46.430148 17170 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 00:53:46.432868 17205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:46.434281 17170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 00:53:46.434428 17170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "a41cd836079747b4a36ca1cfe9a4ed91"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "c3108b854c372596661542d01cd31c6b"
server_key_iv: "82a439186744b5630e7095ecc2393a6a"
server_key_version: "encryptionkey@0"
I20260924 00:53:46.435381 17170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:46.463737 17170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:46.468098 17170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:46.468360 17170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:46.469394 17170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:46.470721 17170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:46.470813 17170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:46.470912 17170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:46.470985 17170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:46.502877 17170 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:38671
I20260924 00:53:46.502949 17318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:38671 every 8 connection(s)
I20260924 00:53:46.504276 17170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:53:46.509562 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 17170
I20260924 00:53:46.510250 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:41031
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:33993
--webserver_port=34207
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:53:46 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:46.528184 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:46.507148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:41007)
0924 00:53:46.507947 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:46.507992 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:46.509810 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:46.511441 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:46.511462 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:46.511980 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:46.512527 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:46.512546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:46.514686 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:46.514690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:46.515907 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:46.515930 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:46.517185 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:46.517190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:46.517420 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:46.518035 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:46.518056 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:46.520213 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:46.523558 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:46.523569 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:46.523586 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:46.524026 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:46.524454 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:46.524459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:46.524463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:46.524659 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:46.525569 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:46.525578 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:46.525933 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 00:53:46.526235 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 00:53:46.526828 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":342,"threads_started":1}
I20260924 00:53:46.528184 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:46.507030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:41007 (local address 127.0.222.62:33993)
0924 00:53:46.507639 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:46.507652 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:46.508116 (+ 464us) server_negotiation.cc:407] Connection header received
0924 00:53:46.510437 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:46.510459 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:46.510916 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:46.511339 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:46.512726 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:46.514534 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:46.516059 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:46.517003 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:46.517186 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:46.520496 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:46.520503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:46.520518 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:46.520557 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:46.523405 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:46.524158 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:46.524165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:46.524170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:46.524338 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:46.524767 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:46.524772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:46.524774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:46.525425 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:46.525630 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:46.526449 (+ 819us) server_negotiation.cc:299] Negotiation successful
0924 00:53:46.526874 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":274,"threads_started":1}
I20260924 00:53:46.531700 17319 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33993
I20260924 00:53:46.532016 17319 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:46.532800 17319 heartbeater.cc:511] Master 127.0.222.62:33993 requested a full tablet report, sending...
I20260924 00:53:46.535338 17132 ts_manager.cc:194] Registered new tserver with Master: a41cd836079747b4a36ca1cfe9a4ed91 (127.0.222.1:38671)
I20260924 00:53:46.537591 17132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:41007
W20260924 00:53:46.850085 17323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:46.850380 17323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:46.850478 17323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:46.858243 17323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:46.858383 17323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:46.858501 17323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:53:46.868643 17323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:41031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=34207
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:46.869966 17323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:46.871301 17323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:46.884542 17331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:46.884613 17329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:46.884585 17328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:46.885897 17323 server_base.cc:1034] running on GCE node
I20260924 00:53:46.886623 17323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:46.887629 17323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:46.888859 17323 hybrid_clock.cc:648] HybridClock initialized: now 1790211226888790 us; error 51 us; skew 500 ppm
I20260924 00:53:46.889189 17323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:46 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:46.892571 17323 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:53:46.894245 17323 webserver.cc:460] Webserver started at http://127.0.222.2:34207/ using document root <none> and password file <none>
I20260924 00:53:46.894955 17323 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:46.895092 17323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:46.900928 17323 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 00:53:46.904145 17338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:46.905772 17323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:53:46.905946 17323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "f69d15061b8c4677a1684de56698e7f0"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "e90a889f8beb23f8876f94cd10ae16d9"
server_key_iv: "eaa9da4f32288474108985bdb413c9b0"
server_key_version: "encryptionkey@0"
I20260924 00:53:46.906968 17323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:46.922719 17323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:46.927520 17323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:46.927790 17323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:46.928781 17323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:46.930148 17323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:46.930236 17323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:46.930341 17323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:46.930406 17323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:46.961294 17323 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41031
I20260924 00:53:46.961345 17453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41031 every 8 connection(s)
I20260924 00:53:46.962767 17323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:53:46.965493 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 17323
I20260924 00:53:46.966220 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:44265
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:33993
--webserver_port=43581
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:53:46 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:46.986459 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:46.967414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:55121 (local address 127.0.222.62:33993)
0924 00:53:46.967567 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:46.967571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:46.967591 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:53:46.969647 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:46.969652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:46.969728 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:46.969901 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:46.971995 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:46.972705 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:46.974959 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:46.975636 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:46.975793 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:46.979736 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:46.979744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:46.979748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:46.979803 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:46.982922 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:46.983818 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:46.983823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:46.983826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:46.983988 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:46.984490 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:46.984495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:46.984498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:46.984870 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:46.985018 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:46.985832 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 00:53:46.986096 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:53:46.987390 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:46.966039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:55121)
0924 00:53:46.966890 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:46.966955 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:46.969289 (+ 2334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:46.970181 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:46.970194 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:46.970947 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:46.971764 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:46.971791 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:46.972868 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:46.972873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:46.974776 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:46.974792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:46.975883 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:46.975889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:46.976112 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:46.976827 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:46.976851 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:46.979424 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:46.983131 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:46.983139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:46.983156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:46.983669 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:46.984134 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:46.984137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:46.984140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:46.984362 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:46.984994 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:46.985001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:46.985422 (+ 421us) client_negotiation.cc:769] Sending connection context
0924 00:53:46.985674 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 00:53:46.986221 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":331,"threads_started":1}
I20260924 00:53:46.988919 17454 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33993
I20260924 00:53:46.989316 17454 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:46.990365 17454 heartbeater.cc:511] Master 127.0.222.62:33993 requested a full tablet report, sending...
I20260924 00:53:46.992200 17132 ts_manager.cc:194] Registered new tserver with Master: f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031)
I20260924 00:53:46.993036 17132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:55121
W20260924 00:53:47.308537 17457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:53:47.308846 17457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:53:47.308987 17457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:53:47.316536 17457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:53:47.316658 17457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:53:47.316812 17457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:53:47.326715 17457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:44265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=43581
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:53:47.328038 17457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:53:47.329401 17457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:53:47.342731 17463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:47.342931 17466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:53:47.342979 17464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:53:47.343514 17457 server_base.cc:1034] running on GCE node
I20260924 00:53:47.344183 17457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:53:47.345168 17457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:53:47.346423 17457 hybrid_clock.cc:648] HybridClock initialized: now 1790211227346355 us; error 57 us; skew 500 ppm
I20260924 00:53:47.346755 17457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:53:47 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:53:47.349987 17457 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:53:47.351435 17457 webserver.cc:460] Webserver started at http://127.0.222.3:43581/ using document root <none> and password file <none>
I20260924 00:53:47.352084 17457 fs_manager.cc:359] Metadata directory not provided
I20260924 00:53:47.352209 17457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:53:47.357767 17457 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 00:53:47.360787 17473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:47.362252 17457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:53:47.362412 17457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "821c0109a8d84220a89985b2e6bc2d1e"
format_stamp: "Formatted at 2026-09-24 00:53:24 on dist-test-slave-rtvz"
server_key: "aeaf5af357b0697b2cc63063c5d91876"
server_key_iv: "b34bae93169bac9e9eceaf963864cd57"
server_key_version: "encryptionkey@0"
I20260924 00:53:47.363431 17457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:53:47.409070 17457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:53:47.413841 17457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:53:47.414116 17457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:53:47.415192 17457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:53:47.416649 17457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:53:47.416767 17457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:47.416887 17457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:53:47.416953 17457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:53:47.448472 17457 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:44265
I20260924 00:53:47.448575 17586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:44265 every 8 connection(s)
I20260924 00:53:47.449940 17457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:53:47.457860 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 17457
Sep 24 00:53:47 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:53:47.472689 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.452539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:43167 (local address 127.0.222.62:33993)
0924 00:53:47.452726 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.452730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.453961 (+ 1231us) server_negotiation.cc:407] Connection header received
0924 00:53:47.455844 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.455848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.455923 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.456016 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:47.457697 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.458707 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.460719 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.461179 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.461330 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.465870 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:47.465875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:47.465878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:47.465910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:47.468840 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.469835 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.469841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.469845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.469994 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.470496 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.470502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.470506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.470902 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:47.471055 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.472079 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.472329 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 00:53:47.473680 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.453002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:43167)
0924 00:53:47.453785 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.453827 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:47.455547 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:47.456232 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:47.456241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:47.456818 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:53:47.457492 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.457515 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.458907 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.458914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.460566 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.460581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.461534 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.461544 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.461855 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.462648 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:53:47.462673 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:53:47.465638 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:53:47.468997 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:47.469008 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:47.469029 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:47.469673 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:47.470110 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:53:47.470115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:53:47.470119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:53:47.470357 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:53:47.471169 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:53:47.471179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:53:47.471656 (+ 477us) client_negotiation.cc:769] Sending connection context
0924 00:53:47.471959 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 00:53:47.472396 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":320,"threads_started":1}
I20260924 00:53:47.475170 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.459019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.0.222.62:33993)
0924 00:53:47.459511 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.459517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.459855 (+ 338us) server_negotiation.cc:407] Connection header received
0924 00:53:47.460023 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.460028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.460098 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.460188 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:47.461977 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.462843 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.464654 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.465219 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.465328 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.468448 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:47.468457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:47.468460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:47.468538 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:47.471815 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.472792 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.472798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.472801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.472933 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.473394 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.473397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.473400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.473678 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:47.474012 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.474662 (+ 650us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.474864 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260924 00:53:47.475291 17587 heartbeater.cc:347] Connected to a master server at 127.0.222.62:33993
I20260924 00:53:47.475621 17587 heartbeater.cc:464] Registering TS with master...
I20260924 00:53:47.476409 17587 heartbeater.cc:511] Master 127.0.222.62:33993 requested a full tablet report, sending...
I20260924 00:53:47.477852 17132 ts_manager.cc:194] Registered new tserver with Master: 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265)
I20260924 00:53:47.478650 17132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:43167
I20260924 00:53:47.490561 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:53:47.497566 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.491538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.0.222.62:33993)
0924 00:53:47.491707 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.491713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.492148 (+ 435us) server_negotiation.cc:407] Connection header received
0924 00:53:47.492379 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.492383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.492559 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.492696 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:53:47.493163 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.493971 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.495453 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.495933 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.496060 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.496578 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:53:47.496811 (+ 233us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 00:53:47.497174 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59722: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 00:53:47.497713 17590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59722: token expired
W20260924 00:53:47.498407 17054 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.222.62:33993)
I20260924 00:53:47.511943 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.498885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.0.222.62:33993)
0924 00:53:47.499098 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.499103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.499123 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:53:47.499290 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.499294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.499384 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.499466 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:47.500857 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.501631 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.503072 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.503481 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.503602 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.506529 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:47.506534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:47.506537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:47.506571 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:47.509147 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.509893 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.509899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.509902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.510037 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.510398 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.510402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.510404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.510793 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:47.511033 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.511373 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.511597 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:53:47.515481 17132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:53:47.522809 17132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:53:47.541715 17319 heartbeater.cc:503] Master 127.0.222.62:33993 was elected leader, sending a full tablet report...
I20260924 00:53:47.546139 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.535028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.0.222.2:41031)
0924 00:53:47.536822 (+ 1794us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.536830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.536859 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:53:47.537011 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.537019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.537298 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.538397 (+ 1099us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:47.538623 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.540264 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.543288 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.544740 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.544884 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.545095 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.545521 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.545752 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1420,"thread_start_us":2097,"threads_started":1}
I20260924 00:53:47.546192 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.535452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.0.222.1:38671)
0924 00:53:47.536500 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.536508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.537178 (+ 670us) server_negotiation.cc:407] Connection header received
0924 00:53:47.537326 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.537333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.537570 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.537745 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:47.538407 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.539896 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.543178 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.544740 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.544878 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.545136 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.545708 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.545898 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":706,"threads_started":1}
I20260924 00:53:47.546236 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.534717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41031 (local address 127.0.0.1:47650)
0924 00:53:47.535602 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.535644 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:47.535825 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:47.537556 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:47.537563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:47.537593 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:47.538124 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.538137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.540450 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.540456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.543094 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.543107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.544990 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.544996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.545238 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.545302 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 00:53:47.545459 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 00:53:47.545689 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":23,"thread_start_us":450,"threads_started":1}
I20260924 00:53:47.547179 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.534826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32782)
0924 00:53:47.536921 (+ 2095us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.536938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:47.537056 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:47.537749 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:47.537753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:47.537777 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:47.538212 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.538225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.540059 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.540063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.542985 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.542999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.545326 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.545331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.545503 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.545537 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 00:53:47.546456 (+ 919us) client_negotiation.cc:240] Negotiation successful
0924 00:53:47.546608 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1970,"mutex_wait_us":4,"thread_start_us":1846,"threads_started":1}
I20260924 00:53:47.549412 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.537047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:44265 (local address 127.0.0.1:54610)
0924 00:53:47.538779 (+ 1732us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.538795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:47.538920 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:47.539939 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:47.539944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:47.539967 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:47.540439 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.540452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.542298 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.542304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.544784 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.544798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.548611 (+ 3813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.548621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.548766 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.548800 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 00:53:47.548888 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:53:47.549032 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"thread_start_us":441,"threads_started":1}
I20260924 00:53:47.549507 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.537653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.0.222.3:44265)
0924 00:53:47.538399 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.538406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.539038 (+ 632us) server_negotiation.cc:407] Connection header received
0924 00:53:47.539129 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.539136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.539488 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.539655 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:47.540629 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.542106 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.544993 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.547384 (+ 2391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.547536 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.547775 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.548972 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.549157 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":440,"threads_started":1}
I20260924 00:53:47.551225 17388 tablet_service.cc:1467] Processing CreateTablet for tablet 0125d066648f40df85ea424136d8e501 (DEFAULT_TABLE table=test-table [id=9b6e945ede80401c82f89478831710b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:47.552168 17254 tablet_service.cc:1467] Processing CreateTablet for tablet 0125d066648f40df85ea424136d8e501 (DEFAULT_TABLE table=test-table [id=9b6e945ede80401c82f89478831710b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:47.552860 17388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0125d066648f40df85ea424136d8e501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:47.552642 17521 tablet_service.cc:1467] Processing CreateTablet for tablet 0125d066648f40df85ea424136d8e501 (DEFAULT_TABLE table=test-table [id=9b6e945ede80401c82f89478831710b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:53:47.553824 17254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0125d066648f40df85ea424136d8e501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:47.554101 17521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0125d066648f40df85ea424136d8e501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:53:47.567651 17603 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: Bootstrap starting.
I20260924 00:53:47.568915 17604 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Bootstrap starting.
I20260924 00:53:47.569779 17605 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: Bootstrap starting.
I20260924 00:53:47.571961 17603 tablet_bootstrap.cc:654] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:47.573144 17604 tablet_bootstrap.cc:654] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:47.573352 17603 log.cc:824] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:47.573585 17605 tablet_bootstrap.cc:654] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: Neither blocks nor log segments found. Creating new log.
I20260924 00:53:47.574331 17604 log.cc:824] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:47.574692 17605 log.cc:824] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:53:47.576841 17603 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: No bootstrap required, opened a new log
I20260924 00:53:47.577184 17603 ts_tablet_manager.cc:1397] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260924 00:53:47.577639 17605 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: No bootstrap required, opened a new log
I20260924 00:53:47.577903 17604 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: No bootstrap required, opened a new log
I20260924 00:53:47.577916 17605 ts_tablet_manager.cc:1397] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260924 00:53:47.578145 17604 ts_tablet_manager.cc:1397] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260924 00:53:47.584839 17604 raft_consensus.cc:348] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.585232 17604 raft_consensus.cc:374] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:47.585357 17604 raft_consensus.cc:729] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41cd836079747b4a36ca1cfe9a4ed91, State: Initialized, Role: FOLLOWER
I20260924 00:53:47.585955 17604 consensus_queue.cc:260] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.587577 17603 raft_consensus.cc:348] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.588066 17603 raft_consensus.cc:374] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:47.588210 17603 raft_consensus.cc:729] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821c0109a8d84220a89985b2e6bc2d1e, State: Initialized, Role: FOLLOWER
I20260924 00:53:47.588160 17605 raft_consensus.cc:348] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.588661 17605 raft_consensus.cc:374] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:53:47.588809 17605 raft_consensus.cc:729] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69d15061b8c4677a1684de56698e7f0, State: Initialized, Role: FOLLOWER
I20260924 00:53:47.589231 17603 consensus_queue.cc:260] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.589430 17604 ts_tablet_manager.cc:1428] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260924 00:53:47.589525 17605 consensus_queue.cc:260] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.590425 17587 heartbeater.cc:503] Master 127.0.222.62:33993 was elected leader, sending a full tablet report...
I20260924 00:53:47.591358 17603 ts_tablet_manager.cc:1428] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260924 00:53:47.592161 17610 raft_consensus.cc:482] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:53:47.592466 17610 raft_consensus.cc:504] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.592926 17605 ts_tablet_manager.cc:1428] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260924 00:53:47.593154 17454 heartbeater.cc:503] Master 127.0.222.62:33993 was elected leader, sending a full tablet report...
I20260924 00:53:47.596295 17610 leader_election.cc:288] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031), a41cd836079747b4a36ca1cfe9a4ed91 (127.0.222.1:38671)
I20260924 00:53:47.601058 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.594723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41031 (local address 127.0.222.3:60235)
0924 00:53:47.595020 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.595041 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:47.595151 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:47.595845 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:47.595850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:47.595878 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:47.596076 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.596089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.597399 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.597404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.599044 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.599055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.600280 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.600286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.600516 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.600549 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:53:47.600651 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 00:53:47.600779 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1}
I20260924 00:53:47.601068 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.595359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:60235 (local address 127.0.222.2:41031)
0924 00:53:47.595539 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.595544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.595562 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:53:47.595628 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.595634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.595705 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.595796 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:47.596237 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.597230 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.599190 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.600125 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.600249 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.600338 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.600687 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.600820 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:53:47.602165 17409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0125d066648f40df85ea424136d8e501" candidate_uuid: "821c0109a8d84220a89985b2e6bc2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69d15061b8c4677a1684de56698e7f0" is_pre_election: true
I20260924 00:53:47.602649 17409 raft_consensus.cc:2454] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821c0109a8d84220a89985b2e6bc2d1e in term 0.
I20260924 00:53:47.602826 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.596652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.222.3:55019)
0924 00:53:47.597362 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.597379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:53:47.597481 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:53:47.597815 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:53:47.597820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:53:47.597846 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:53:47.598260 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.598273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.599236 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.599239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.600687 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:53:47.600700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.602264 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.602269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:53:47.602381 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.602402 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 00:53:47.602481 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 00:53:47.602578 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":362,"threads_started":1}
I20260924 00:53:47.602883 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.597348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:55019 (local address 127.0.222.1:38671)
0924 00:53:47.597532 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.597538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.597556 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:53:47.597614 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.597618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.597686 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.597778 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:53:47.598430 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.599114 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.601007 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.601980 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.602058 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.602125 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.602525 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.602653 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:53:47.603384 17475 leader_election.cc:302] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 821c0109a8d84220a89985b2e6bc2d1e, f69d15061b8c4677a1684de56698e7f0; no voters:
I20260924 00:53:47.603740 17610 raft_consensus.cc:2781] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:53:47.603870 17610 raft_consensus.cc:482] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:53:47.603965 17610 raft_consensus.cc:3037] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:47.603775 17274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0125d066648f40df85ea424136d8e501" candidate_uuid: "821c0109a8d84220a89985b2e6bc2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" is_pre_election: true
I20260924 00:53:47.604251 17274 raft_consensus.cc:2454] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821c0109a8d84220a89985b2e6bc2d1e in term 0.
I20260924 00:53:47.607960 17610 raft_consensus.cc:504] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.608620 17610 leader_election.cc:288] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [CANDIDATE]: Term 1 election: Requested vote from peers f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031), a41cd836079747b4a36ca1cfe9a4ed91 (127.0.222.1:38671)
I20260924 00:53:47.609141 17409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0125d066648f40df85ea424136d8e501" candidate_uuid: "821c0109a8d84220a89985b2e6bc2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69d15061b8c4677a1684de56698e7f0"
I20260924 00:53:47.609316 17274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0125d066648f40df85ea424136d8e501" candidate_uuid: "821c0109a8d84220a89985b2e6bc2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41cd836079747b4a36ca1cfe9a4ed91"
I20260924 00:53:47.609409 17409 raft_consensus.cc:3037] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:47.609548 17274 raft_consensus.cc:3037] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:53:47.612677 17274 raft_consensus.cc:2454] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821c0109a8d84220a89985b2e6bc2d1e in term 1.
I20260924 00:53:47.612677 17409 raft_consensus.cc:2454] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821c0109a8d84220a89985b2e6bc2d1e in term 1.
I20260924 00:53:47.613095 17475 leader_election.cc:302] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 821c0109a8d84220a89985b2e6bc2d1e, f69d15061b8c4677a1684de56698e7f0; no voters:
I20260924 00:53:47.613394 17610 raft_consensus.cc:2781] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:53:47.613858 17610 raft_consensus.cc:686] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 1 LEADER]: Becoming Leader. State: Replica: 821c0109a8d84220a89985b2e6bc2d1e, State: Running, Role: LEADER
I20260924 00:53:47.614333 17610 consensus_queue.cc:237] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:53:47.619160 17132 catalog_manager.cc:5495] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3). New cstate: current_term: 1 leader_uuid: "821c0109a8d84220a89985b2e6bc2d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } health_report { overall_health: HEALTHY } } }
I20260924 00:53:47.649183 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.643586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.0.222.3:44265)
0924 00:53:47.643781 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.643785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.643804 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:53:47.643896 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.643899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.643993 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.644114 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:53:47.644516 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.645310 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.646914 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.647339 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.647440 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.648028 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:53:47.648552 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:53:47.648662 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.648839 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.648916 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:53:47.657564 17409 raft_consensus.cc:1261] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 1 FOLLOWER]: Refusing update from remote peer 821c0109a8d84220a89985b2e6bc2d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:53:47.657572 17274 raft_consensus.cc:1261] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Refusing update from remote peer 821c0109a8d84220a89985b2e6bc2d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:53:47.658479 17613 consensus_queue.cc:1035] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:53:47.658823 17610 consensus_queue.cc:1035] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:53:47.687529 17132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:53:47.687839 17132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:53:47.691876 17132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Sending DeleteTablet for 3 replicas of tablet 0125d066648f40df85ea424136d8e501
I20260924 00:53:47.693149 17388 tablet_service.cc:1514] Processing DeleteTablet for tablet 0125d066648f40df85ea424136d8e501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:47 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:47650
I20260924 00:53:47.693250 17254 tablet_service.cc:1514] Processing DeleteTablet for tablet 0125d066648f40df85ea424136d8e501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:47 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:32782
I20260924 00:53:47.693373 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 17170
I20260924 00:53:47.694320 17521 tablet_service.cc:1514] Processing DeleteTablet for tablet 0125d066648f40df85ea424136d8e501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:53:47 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:54610
I20260924 00:53:47.723404 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.695105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.0.222.1:38671)
0924 00:53:47.695331 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.695337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.695357 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:53:47.695419 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.695424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.695529 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.695634 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:47.697774 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.699364 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.703050 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.703764 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.703906 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.712032 (+ 8126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:47.712059 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:47.712084 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:47.712147 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:47.716567 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.717942 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.717956 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.717960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.718170 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.719815 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.719821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.719825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.720372 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:47.720721 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.722654 (+ 1933us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.722962 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 00:53:47.725493 17314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:47.730731 17625 tablet_replica.cc:330] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: stopping tablet replica
I20260924 00:53:47.731454 17627 tablet_replica.cc:330] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: stopping tablet replica
I20260924 00:53:47.732236 17627 raft_consensus.cc:2229] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:53:47.732770 17625 raft_consensus.cc:2229] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:53:47.733179 17627 raft_consensus.cc:2258] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:47.733556 17625 raft_consensus.cc:2258] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:47.735610 17627 ts_tablet_manager.cc:1905] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:47.735682 17625 ts_tablet_manager.cc:1905] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:53:47.743148 17625 ts_tablet_manager.cc:1918] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:47.743429 17625 log.cc:1196] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/0125d066648f40df85ea424136d8e501
I20260924 00:53:47.743448 17627 ts_tablet_manager.cc:1918] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:53:47.743772 17627 log.cc:1196] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/0125d066648f40df85ea424136d8e501
I20260924 00:53:47.744072 17625 ts_tablet_manager.cc:1939] T 0125d066648f40df85ea424136d8e501 P f69d15061b8c4677a1684de56698e7f0: Deleting consensus metadata
I20260924 00:53:47.744406 17627 ts_tablet_manager.cc:1939] T 0125d066648f40df85ea424136d8e501 P 821c0109a8d84220a89985b2e6bc2d1e: Deleting consensus metadata
I20260924 00:53:47.746235 17119 catalog_manager.cc:4841] TS 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265): tablet 0125d066648f40df85ea424136d8e501 (table test-table [id=9b6e945ede80401c82f89478831710b9]) successfully deleted
I20260924 00:53:47.746213 17117 catalog_manager.cc:4841] TS f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031): tablet 0125d066648f40df85ea424136d8e501 (table test-table [id=9b6e945ede80401c82f89478831710b9]) successfully deleted
I20260924 00:53:47.827834 17626 tablet_replica.cc:330] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: stopping tablet replica
I20260924 00:53:47.828217 17626 raft_consensus.cc:2229] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:53:47.828642 17626 raft_consensus.cc:2258] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:53:47.829538 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 17170
W20260924 00:53:47.844358 17119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:38671 (error 108)
I20260924 00:53:47.844566 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 17323
I20260924 00:53:47.847926 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.847404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32808)
0924 00:53:47.847602 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.847732 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 00:53:47.848248 17119 catalog_manager.cc:4584] TS a41cd836079747b4a36ca1cfe9a4ed91 (127.0.222.1:38671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0125d066648f40df85ea424136d8e501: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
I20260924 00:53:47.864168 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.845671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.0.222.2:41031)
0924 00:53:47.845869 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:47.845876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:47.845896 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:53:47.845999 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:47.846005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:47.846112 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:47.846230 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:47.847961 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.848866 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.850268 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:47.850666 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:47.850789 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:47.857439 (+ 6650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:47.857459 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:47.857476 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:47.857519 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:47.860664 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.861753 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.861761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.861764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.861907 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:47.862315 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:47.862318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:47.862320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:47.862742 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:47.863036 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:47.863445 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 00:53:47.863689 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:53:47.865068 17449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:47.902146 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.901667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32818)
0924 00:53:47.901887 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.901988 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 00:53:47.903235 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.902883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32826)
0924 00:53:47.903039 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.903112 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 00:53:47.963945 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.963298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32832)
0924 00:53:47.963590 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.963720 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 00:53:47.965291 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:47.964824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32834)
0924 00:53:47.965001 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:47.965100 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 00:53:47.989251 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 17323
I20260924 00:53:48.005147 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 17457
I20260924 00:53:48.022727 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:48.006038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.222.3:44265)
0924 00:53:48.006279 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 00:53:48.006286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:53:48.006381 (+ 95us) server_negotiation.cc:407] Connection header received
0924 00:53:48.006552 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:53:48.006558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:53:48.006652 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:53:48.006772 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:53:48.008270 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:48.009501 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:48.010861 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:53:48.011547 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:53:48.011697 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:53:48.015216 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:53:48.015243 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:53:48.015266 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:53:48.015323 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:53:48.019441 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:48.020308 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:48.020318 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:48.020322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:48.020529 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:53:48.020955 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:53:48.020960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:53:48.020963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:53:48.021451 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:53:48.021761 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 00:53:48.022077 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 00:53:48.022306 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 00:53:48.023499 17582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:48.046154 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:48.045604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32850)
0924 00:53:48.045888 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:48.045999 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 00:53:48.047384 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:48.046957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32858)
0924 00:53:48.047177 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:48.047255 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 00:53:48.145581 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 17457
I20260924 00:53:48.162883 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 17099
I20260924 00:53:48.164029 17163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:53:48.275219 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:53:48.274623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:38671 (local address 127.0.0.1:32868)
0924 00:53:48.274898 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 00:53:48.275021 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:38671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 00:53:48.275205 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 17099
I20260924 00:54:08.293386 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33993
--webserver_interface=127.0.222.62
--webserver_port=44755
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:08.624281 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:08.624619 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:08.624748 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:08.632158 17633 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:54:08.632283 17633 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:54:08.632339 17633 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:54:08.632409 17633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:08.632478 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:08.632576 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:08.632634 17633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:08.632705 17633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:08.643013 17633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=44755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:08.644315 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:08.645718 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:08.657704 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:08.657649 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:08.657673 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:08.658473 17633 server_base.cc:1034] running on GCE node
I20260924 00:54:08.659224 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:08.660630 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:08.661850 17633 hybrid_clock.cc:648] HybridClock initialized: now 1790211248661818 us; error 92 us; skew 500 ppm
I20260924 00:54:08.662174 17633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:08 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:08.669075 17633 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:54:08.670830 17633 webserver.cc:460] Webserver started at http://127.0.222.62:44755/ using document root <none> and password file <none>
I20260924 00:54:08.671588 17633 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:08.671686 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:08.677692 17633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 00:54:08.680755 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:08.682482 17633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:08.682651 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "c0b025243c3549959ef7293ab5d8a5b2"
format_stamp: "Formatted at 2026-09-24 00:53:22 on dist-test-slave-rtvz"
server_key: "2e885ed48d2329184193bcc8057bdc01"
server_key_iv: "140128d8351d0b354303ca5d6b7da27c"
server_key_version: "encryptionkey@0"
I20260924 00:54:08.683640 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:08.708666 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:08.713387 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:08.713743 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:08.732012 17633 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33993
I20260924 00:54:08.732020 17700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33993 every 8 connection(s)
I20260924 00:54:08.733651 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:08.737479 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 17633
I20260924 00:54:08.738538 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:38671
--local_ip_for_outbound_sockets=127.0.222.1
--tserver_master_addrs=127.0.222.62:33993
--webserver_port=44137
--webserver_interface=127.0.222.1
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:54:08.742435 17701 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:54:08.747107 17701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Bootstrap starting.
I20260924 00:54:08.767586 17701 log.cc:824] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:08.783999 17701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:54:08.784554 17701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Bootstrap complete.
I20260924 00:54:08.793803 17701 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:54:08.794739 17701 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b025243c3549959ef7293ab5d8a5b2, State: Initialized, Role: FOLLOWER
I20260924 00:54:08.795360 17701 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:54:08.795571 17701 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:08.795681 17701 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:08.795850 17701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:54:08.799691 17701 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:54:08.800186 17701 leader_election.cc:302] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0b025243c3549959ef7293ab5d8a5b2; no voters:
I20260924 00:54:08.800813 17701 leader_election.cc:288] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 00:54:08.800983 17705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:54:08.802080 17705 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [term 3 LEADER]: Becoming Leader. State: Replica: c0b025243c3549959ef7293ab5d8a5b2, State: Running, Role: LEADER
I20260924 00:54:08.802779 17705 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } }
I20260924 00:54:08.803157 17701 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:08.805415 17706 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c0b025243c3549959ef7293ab5d8a5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } } }
I20260924 00:54:08.805773 17706 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:08.806277 17707 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0b025243c3549959ef7293ab5d8a5b2. Latest consensus state: current_term: 3 leader_uuid: "c0b025243c3549959ef7293ab5d8a5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b025243c3549959ef7293ab5d8a5b2" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33993 } } }
I20260924 00:54:08.806632 17707 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:08.814721 17711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:08.824008 17711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:08.825742 17711 catalog_manager.cc:1259] Loaded cluster ID: fa44cbca641b44a68353982f5d5ade5a
I20260924 00:54:08.825906 17711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:08.830495 17711 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:08.832612 17711 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Loaded TSK: 0
I20260924 00:54:08.832897 17711 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c0b025243c3549959ef7293ab5d8a5b2: Loaded TSK: 1
F20260924 00:54:08.834228 17711 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f17b19769ad google::LogMessage::Fail() at ??:0
@ 0x7f17b197988a google::LogMessage::SendToLog() at ??:0
@ 0x7f17b19764de google::LogMessage::Flush() at ??:0
@ 0x7f17b1977a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260924 00:54:08.843127 17722 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f17b56da7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f17b56c973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f17c73a51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f17c73a4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f17c74302a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f17c74300c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f17c747a5e9 std::function<>::operator()() at ??:0
@ 0x7f17c73a8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f17c73a6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f17c742e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f17c742e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f17b45c2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f17b45c591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f17b45c5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f17b45a0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f17b787a6db start_thread at ??:0
@ 0x7f17b081061f clone at ??:0
W20260924 00:54:09.150689 17703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:09.150998 17703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:09.151094 17703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:09.158955 17703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:09.159078 17703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:09.159192 17703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:54:09.169373 17703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:38671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=44137
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:09.170619 17703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:09.171886 17703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:09.185232 17749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:09.185148 17751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:09.185254 17748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:09.186014 17703 server_base.cc:1034] running on GCE node
I20260924 00:54:09.186693 17703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:09.187707 17703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:09.188982 17703 hybrid_clock.cc:648] HybridClock initialized: now 1790211249188930 us; error 41 us; skew 500 ppm
I20260924 00:54:09.189363 17703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:09 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:09.192637 17703 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:54:09.194315 17703 webserver.cc:460] Webserver started at http://127.0.222.1:44137/ using document root <none> and password file <none>
I20260924 00:54:09.195020 17703 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:09.195128 17703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:09.200721 17703 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 00:54:09.203778 17758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:09.205281 17703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 00:54:09.205427 17703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "a41cd836079747b4a36ca1cfe9a4ed91"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "c3108b854c372596661542d01cd31c6b"
server_key_iv: "82a439186744b5630e7095ecc2393a6a"
server_key_version: "encryptionkey@0"
I20260924 00:54:09.206473 17703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:09.219949 17703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:09.224014 17703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:09.224241 17703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:09.225216 17703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:09.227231 17765 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 00:54:09.234203 17703 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:54:09.234349 17703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.001s
I20260924 00:54:09.234489 17703 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:54:09.237895 17703 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:54:09.238000 17703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.001s
I20260924 00:54:09.238394 17765 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Bootstrap starting.
I20260924 00:54:09.247560 17765 log.cc:824] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:09.263115 17765 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:54:09.264065 17765 tablet_bootstrap.cc:492] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Bootstrap complete.
I20260924 00:54:09.264672 17765 ts_tablet_manager.cc:1397] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.005s
I20260924 00:54:09.273566 17703 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:38671
I20260924 00:54:09.273584 17873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:38671 every 8 connection(s)
I20260924 00:54:09.274865 17703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:54:09.274046 17765 raft_consensus.cc:348] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:54:09.275012 17765 raft_consensus.cc:729] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41cd836079747b4a36ca1cfe9a4ed91, State: Initialized, Role: FOLLOWER
I20260924 00:54:09.275739 17765 consensus_queue.cc:260] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:54:09.277679 17765 ts_tablet_manager.cc:1428] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 00:54:09.279158 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 17703
I20260924 00:54:09.279716 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:09.277815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:47427)
0924 00:54:09.278618 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:09.278826 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":303,"threads_started":1}
I20260924 00:54:09.279771 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:41031
--local_ip_for_outbound_sockets=127.0.222.2
--tserver_master_addrs=127.0.222.62:33993
--webserver_port=34207
--webserver_interface=127.0.222.2
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:09.280376 17874 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:33993 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:33993: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
I20260924 00:54:09.281158 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:09.280850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:51341)
0924 00:54:09.280995 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:09.281063 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 00:54:09.281980 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:09.281653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:57671)
0924 00:54:09.281787 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:09.281858 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 00:54:09.282191 17874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:54:09.606616 17878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:09.607077 17878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:09.607223 17878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:09.614662 17878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:09.614787 17878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:09.614905 17878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:54:09.624819 17878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:41031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=34207
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:09.626139 17878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:09.627434 17878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:09.640489 17884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:09.640518 17886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:09.640573 17883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:09.641297 17878 server_base.cc:1034] running on GCE node
I20260924 00:54:09.641913 17878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:09.642864 17878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:09.644083 17878 hybrid_clock.cc:648] HybridClock initialized: now 1790211249644023 us; error 49 us; skew 500 ppm
I20260924 00:54:09.644383 17878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:09 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:09.647554 17878 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:54:09.649106 17878 webserver.cc:460] Webserver started at http://127.0.222.2:34207/ using document root <none> and password file <none>
I20260924 00:54:09.649811 17878 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:09.649935 17878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:09.655412 17878 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 00:54:09.658088 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:09.659435 17878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 00:54:09.659582 17878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "f69d15061b8c4677a1684de56698e7f0"
format_stamp: "Formatted at 2026-09-24 00:53:23 on dist-test-slave-rtvz"
server_key: "e90a889f8beb23f8876f94cd10ae16d9"
server_key_iv: "eaa9da4f32288474108985bdb413c9b0"
server_key_version: "encryptionkey@0"
I20260924 00:54:09.660576 17878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:09.690045 17878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:09.694685 17878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:09.694973 17878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:09.695994 17878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:09.697477 17878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:09.697587 17878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:09.697701 17878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:09.697767 17878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:09.729729 17878 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:41031
I20260924 00:54:09.729787 18006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:41031 every 8 connection(s)
I20260924 00:54:09.731151 17878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:54:09.733588 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 17878
I20260924 00:54:09.734170 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:44265
--local_ip_for_outbound_sockets=127.0.222.3
--tserver_master_addrs=127.0.222.62:33993
--webserver_port=43581
--webserver_interface=127.0.222.3
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:54:09.736388 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:09.734134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:54823)
0924 00:54:09.734990 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:09.735254 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":306,"threads_started":1}
W20260924 00:54:09.737159 18007 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:33993 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:33993: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
I20260924 00:54:09.738132 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:09.737793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:36203)
0924 00:54:09.737953 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:09.738024 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 00:54:09.738945 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:09.738617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:52843)
0924 00:54:09.738744 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:09.738829 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 00:54:09.739210 18007 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:54:10.065927 18010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:10.066233 18010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:10.066332 18010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:10.073727 18010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:10.073848 18010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:10.073971 18010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:54:10.084049 18010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:44265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=43581
--tserver_master_addrs=127.0.222.62:33993
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:10.085428 18010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:10.086777 18010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:10.100543 18015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:10.100517 18016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:10.100502 18018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:10.107017 18010 server_base.cc:1034] running on GCE node
I20260924 00:54:10.107699 18010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:10.108757 18010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:10.109978 18010 hybrid_clock.cc:648] HybridClock initialized: now 1790211250109922 us; error 27 us; skew 500 ppm
I20260924 00:54:10.110284 18010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:10 dist-test-slave-rtvz krb5kdc[16545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:10.113467 18010 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:54:10.115031 18010 webserver.cc:460] Webserver started at http://127.0.222.3:43581/ using document root <none> and password file <none>
I20260924 00:54:10.115729 18010 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:10.115839 18010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:10.121665 18010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 00:54:10.124462 18025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:10.125963 18010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:10.126109 18010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "821c0109a8d84220a89985b2e6bc2d1e"
format_stamp: "Formatted at 2026-09-24 00:53:24 on dist-test-slave-rtvz"
server_key: "aeaf5af357b0697b2cc63063c5d91876"
server_key_iv: "b34bae93169bac9e9eceaf963864cd57"
server_key_version: "encryptionkey@0"
I20260924 00:54:10.127103 18010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:10.143149 18010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:10.147368 18010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:10.147598 18010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:10.148594 18010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:10.149986 18010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:10.150068 18010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:10.150146 18010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:10.150200 18010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:10.181702 18010 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:44265
I20260924 00:54:10.181881 18138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:44265 every 8 connection(s)
I20260924 00:54:10.183128 18010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:54:10.188153 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.186144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:36601)
0924 00:54:10.186898 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:10.187125 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":289,"threads_started":1}
W20260924 00:54:10.188844 18139 heartbeater.cc:644] Failed to heartbeat to 127.0.222.62:33993 (0 consecutive failures): Network error: Failed to ping master at 127.0.222.62:33993: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
I20260924 00:54:10.189509 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 18010
I20260924 00:54:10.189841 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.189446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:44301)
0924 00:54:10.189618 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:10.189706 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 00:54:10.190675 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.190371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:59699)
0924 00:54:10.190488 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:10.190561 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 00:54:10.191097 18139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:54:10.283506 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.282684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:36109)
0924 00:54:10.283176 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:10.283270 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":236,"threads_started":1}
I20260924 00:54:10.740629 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.739815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:56019)
0924 00:54:10.740300 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:10.740400 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":234,"threads_started":1}
I20260924 00:54:10.850791 18145 raft_consensus.cc:482] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:10.851142 18145 raft_consensus.cc:504] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:54:10.852720 18145 leader_election.cc:288] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031), 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265)
Sep 24 00:54:10 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
Sep 24 00:54:10 dist-test-slave-rtvz krb5kdc[16545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:54:10.888249 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.853156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:41031 (local address 127.0.222.1:45991)
0924 00:54:10.866593 (+ 13437us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:10.866616 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:10.867234 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:10.870779 (+ 3545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:10.870790 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:10.871670 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:10.872460 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:10.872479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:10.874100 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:10.874105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:10.875572 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:10.875586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:10.876575 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:10.876585 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:10.876858 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:10.877640 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:10.877657 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:10.880401 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:10.883899 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:10.883910 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:10.883922 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:10.884421 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:10.884882 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:10.884887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:10.884890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:10.885167 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:10.886401 (+ 1234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:10.886409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:10.886905 (+ 496us) client_negotiation.cc:769] Sending connection context
0924 00:54:10.887215 (+ 310us) client_negotiation.cc:240] Negotiation successful
0924 00:54:10.887799 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13343,"mutex_wait_us":129,"spinlock_wait_cycles":18432,"thread_start_us":13071,"threads_started":1}
I20260924 00:54:10.888597 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.853396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:39711 (local address 127.0.222.3:44265)
0924 00:54:10.866193 (+ 12797us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:10.866209 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:10.866785 (+ 576us) server_negotiation.cc:407] Connection header received
0924 00:54:10.869670 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:10.869683 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:10.870204 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:10.870591 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:10.872654 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:10.873898 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:10.875717 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:10.876357 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:10.876541 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:10.880379 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:10.880396 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:10.880413 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:10.880452 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:10.883741 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:10.884579 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:10.884588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:10.884595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:10.884811 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:10.885327 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:10.885331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:10.885335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:10.886125 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:10.886313 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:10.887696 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0924 00:54:10.888167 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12624,"thread_start_us":12474,"threads_started":1}
I20260924 00:54:10.888602 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.853451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:45991 (local address 127.0.222.2:41031)
0924 00:54:10.865917 (+ 12466us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:10.865940 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:10.866783 (+ 843us) server_negotiation.cc:407] Connection header received
0924 00:54:10.869670 (+ 2887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:10.869683 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:10.870204 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:10.870591 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:10.872669 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:10.873939 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:10.875713 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:10.876357 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:10.876533 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:10.880661 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:10.880669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:10.880683 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:10.880721 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:10.883763 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:10.884558 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:10.884566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:10.884571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:10.884767 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:10.885338 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:10.885343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:10.885346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:10.886126 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:10.886313 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:10.887591 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0924 00:54:10.888219 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12211,"thread_start_us":11998,"threads_started":1}
I20260924 00:54:10.889477 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:10.853127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:44265 (local address 127.0.222.1:39711)
0924 00:54:10.866560 (+ 13433us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:10.866616 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:10.867245 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:10.871192 (+ 3947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:10.871197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:10.871755 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:10.872458 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:10.872478 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:10.874098 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:10.874102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:10.875576 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:10.875589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:10.876611 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:10.876619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:10.876862 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:10.877597 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:10.877633 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:10.880071 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:10.883896 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:10.883906 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:10.883922 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:10.884427 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:10.884954 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:10.884959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:10.884962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:10.885165 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:10.886404 (+ 1239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:10.886412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:10.886912 (+ 500us) client_negotiation.cc:769] Sending connection context
0924 00:54:10.887214 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 00:54:10.889063 (+ 1849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13258,"mutex_wait_us":1511,"spinlock_wait_cycles":84736,"thread_start_us":13247,"threads_started":1}
W20260924 00:54:10.892481 17760 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
W20260924 00:54:10.892982 17762 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
I20260924 00:54:10.893138 17762 leader_election.cc:302] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a41cd836079747b4a36ca1cfe9a4ed91; no voters: 821c0109a8d84220a89985b2e6bc2d1e, f69d15061b8c4677a1684de56698e7f0
I20260924 00:54:10.893915 18145 raft_consensus.cc:2726] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:54:11.192976 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:11.191797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:40993)
0924 00:54:11.192482 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:11.192653 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":355,"threads_started":1}
I20260924 00:54:11.284993 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:11.284527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:34467)
0924 00:54:11.284718 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:11.284830 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 00:54:11.742391 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:11.741647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:50323)
0924 00:54:11.742133 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:11.742227 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":300,"threads_started":1}
I20260924 00:54:12.194765 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:12.193998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:34625)
0924 00:54:12.194508 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:12.194601 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260924 00:54:12.286645 18153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:12.285897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:53719)
0924 00:54:12.286398 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:12.286476 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":290,"threads_started":1}
I20260924 00:54:12.743943 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:12.743238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:41093)
0924 00:54:12.743676 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:12.743761 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":232,"threads_started":1}
I20260924 00:54:13.137887 18155 raft_consensus.cc:482] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:13.138075 18155 raft_consensus.cc:504] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:54:13.138674 18155 leader_election.cc:288] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031), 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265)
W20260924 00:54:13.139690 17762 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
W20260924 00:54:13.139848 17760 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
I20260924 00:54:13.139998 17760 leader_election.cc:302] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a41cd836079747b4a36ca1cfe9a4ed91; no voters: 821c0109a8d84220a89985b2e6bc2d1e, f69d15061b8c4677a1684de56698e7f0
I20260924 00:54:13.140234 18155 raft_consensus.cc:2726] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:54:13.196396 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:13.195675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:45709)
0924 00:54:13.196140 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:13.196221 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":247,"threads_started":1}
I20260924 00:54:13.288362 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:13.287510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:41023)
0924 00:54:13.288050 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:13.288148 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":224,"threads_started":1}
I20260924 00:54:13.745744 18158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:13.744897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:43391)
0924 00:54:13.745453 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:13.745554 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":299,"threads_started":1}
I20260924 00:54:14.198342 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:14.197453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:41933)
0924 00:54:14.198014 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:14.198105 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":244,"threads_started":1}
I20260924 00:54:14.290526 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:14.289658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:56133)
0924 00:54:14.290239 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:14.290331 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":327,"threads_started":1}
I20260924 00:54:14.747502 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:14.746702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:51169)
0924 00:54:14.747244 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:14.747333 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":299,"threads_started":1}
I20260924 00:54:15.200093 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:15.199300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:52125)
0924 00:54:15.199828 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:15.199915 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":298,"threads_started":1}
I20260924 00:54:15.292250 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:15.291509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:54739)
0924 00:54:15.291980 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:15.292084 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260924 00:54:15.749248 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:15.748444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:47085)
0924 00:54:15.748935 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:15.749026 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":275,"threads_started":1}
I20260924 00:54:16.129947 18165 raft_consensus.cc:482] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:16.130126 18165 raft_consensus.cc:504] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:54:16.130788 18165 leader_election.cc:288] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031), 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265)
W20260924 00:54:16.131773 17760 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
W20260924 00:54:16.131942 17762 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
I20260924 00:54:16.132099 17762 leader_election.cc:302] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a41cd836079747b4a36ca1cfe9a4ed91; no voters: 821c0109a8d84220a89985b2e6bc2d1e, f69d15061b8c4677a1684de56698e7f0
I20260924 00:54:16.132445 18165 raft_consensus.cc:2726] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:54:16.201864 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:16.201008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:38973)
0924 00:54:16.201605 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:16.201685 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":333,"threads_started":1}
I20260924 00:54:16.294055 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:16.293181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:59877)
0924 00:54:16.293735 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:16.293856 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":254,"threads_started":1}
I20260924 00:54:16.751160 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:16.750297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:39199)
0924 00:54:16.750848 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:16.750959 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":238,"threads_started":1}
I20260924 00:54:17.203610 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:17.202771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:49087)
0924 00:54:17.203306 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:17.203420 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":254,"threads_started":1}
I20260924 00:54:17.295635 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:17.294906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:44501)
0924 00:54:17.295384 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:17.295470 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260924 00:54:17.752885 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:17.752154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:52097)
0924 00:54:17.752601 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:17.752699 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260924 00:54:18.205413 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:18.204601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:44987)
0924 00:54:18.205072 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:18.205154 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":260,"threads_started":1}
I20260924 00:54:18.297513 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:18.296619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:45865)
0924 00:54:18.297184 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:18.297317 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":314,"threads_started":1}
I20260924 00:54:18.754709 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:18.753943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:34925)
0924 00:54:18.754439 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:18.754546 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":232,"threads_started":1}
I20260924 00:54:19.127127 18175 raft_consensus.cc:482] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:19.127314 18175 raft_consensus.cc:504] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:54:19.128036 18175 leader_election.cc:288] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031), 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265)
W20260924 00:54:19.129086 17760 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
W20260924 00:54:19.129316 17762 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
I20260924 00:54:19.129463 17762 leader_election.cc:302] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a41cd836079747b4a36ca1cfe9a4ed91; no voters: 821c0109a8d84220a89985b2e6bc2d1e, f69d15061b8c4677a1684de56698e7f0
I20260924 00:54:19.129736 18175 raft_consensus.cc:2726] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:54:19.207207 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:19.206462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:60815)
0924 00:54:19.206963 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:19.207050 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
I20260924 00:54:19.299328 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:19.298525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:34145)
0924 00:54:19.299036 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:19.299118 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":247,"threads_started":1}
I20260924 00:54:19.756524 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:19.755718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:47139)
0924 00:54:19.756229 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:19.756317 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":267,"threads_started":1}
I20260924 00:54:20.209024 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:20.208268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:36357)
0924 00:54:20.208752 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:20.208839 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
I20260924 00:54:20.301136 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:20.300267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:43183)
0924 00:54:20.300779 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:20.300938 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":261,"threads_started":1}
I20260924 00:54:20.758394 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:20.757610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:45035)
0924 00:54:20.758113 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:20.758218 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":265,"threads_started":1}
I20260924 00:54:21.211071 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:21.210113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:43077)
0924 00:54:21.210641 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:21.210754 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260924 00:54:21.303052 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:21.302241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:39577)
0924 00:54:21.302715 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:21.302804 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260924 00:54:21.760344 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:21.759570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:43433)
0924 00:54:21.760068 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:21.760165 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":252,"threads_started":1}
I20260924 00:54:22.212898 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:22.212120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:50995)
0924 00:54:22.212622 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:22.212714 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260924 00:54:22.304826 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:22.304051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:47757)
0924 00:54:22.304546 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:22.304639 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":247,"threads_started":1}
I20260924 00:54:22.762099 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:22.761361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:34235)
0924 00:54:22.761828 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:22.761923 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260924 00:54:23.214994 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:23.214020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:55643)
0924 00:54:23.214554 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:23.214687 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":260,"threads_started":1}
I20260924 00:54:23.306559 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:23.305777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:42149)
0924 00:54:23.306293 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:23.306381 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":297,"threads_started":1}
I20260924 00:54:23.763972 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:23.763108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:45477)
0924 00:54:23.763598 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:23.763711 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":232,"threads_started":1}
I20260924 00:54:24.051292 18191 raft_consensus.cc:482] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:24.051492 18191 raft_consensus.cc:504] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f69d15061b8c4677a1684de56698e7f0" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 41031 } } peers { permanent_uuid: "a41cd836079747b4a36ca1cfe9a4ed91" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 38671 } } peers { permanent_uuid: "821c0109a8d84220a89985b2e6bc2d1e" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44265 } }
I20260924 00:54:24.052201 18191 leader_election.cc:288] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031), 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265)
W20260924 00:54:24.053115 17760 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f69d15061b8c4677a1684de56698e7f0 (127.0.222.2:41031): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
W20260924 00:54:24.053306 17762 leader_election.cc:341] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 821c0109a8d84220a89985b2e6bc2d1e (127.0.222.3:44265): Not found: Tablet not found: 0125d066648f40df85ea424136d8e501
I20260924 00:54:24.053432 17762 leader_election.cc:302] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a41cd836079747b4a36ca1cfe9a4ed91; no voters: 821c0109a8d84220a89985b2e6bc2d1e, f69d15061b8c4677a1684de56698e7f0
I20260924 00:54:24.053692 18191 raft_consensus.cc:2726] T 0125d066648f40df85ea424136d8e501 P a41cd836079747b4a36ca1cfe9a4ed91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:54:24.216711 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:24.215908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:52341)
0924 00:54:24.216429 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:24.216510 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":248,"threads_started":1}
I20260924 00:54:24.308673 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:24.307775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.1:56673)
0924 00:54:24.308316 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:24.308437 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":288,"threads_started":1}
I20260924 00:54:24.765865 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:24.765089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.2:50939)
0924 00:54:24.765584 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:24.765671 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":266,"threads_started":1}
I20260924 00:54:25.193392 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 17703
I20260924 00:54:25.214021 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:25.194781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.222.1:38671)
0924 00:54:25.195319 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:25.195333 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:25.195361 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:54:25.195427 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:25.195434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:25.195653 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:25.195818 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:25.197317 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:25.198289 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:25.199521 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:25.200067 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:25.200181 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:25.207581 (+ 7400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:25.207595 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:25.207611 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:25.207654 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:25.210774 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:25.211605 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:25.211612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:25.211614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:25.211771 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:25.212193 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:25.212200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:25.212202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:25.212629 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:25.212943 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:25.213361 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 00:54:25.213670 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260924 00:54:25.215341 17869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:25.218976 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:25.217914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:33993 (local address 127.0.222.3:58453)
0924 00:54:25.218515 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:25.218710 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.62:33993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":258,"threads_started":1}
I20260924 00:54:25.306871 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 17703
I20260924 00:54:25.321720 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 17878
I20260924 00:54:25.335420 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:25.322439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.0.222.2:41031)
0924 00:54:25.322881 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:25.322886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:25.322903 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:54:25.322958 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:25.322962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:25.323021 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:25.323124 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:25.324484 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:25.325323 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:25.326498 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:25.326897 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:25.326986 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:25.329897 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:25.329902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:25.329905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:25.329943 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:25.332670 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:25.333619 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:25.333624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:25.333626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:25.333775 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:25.334179 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:25.334182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:25.334184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:25.334469 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:25.334698 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:25.334928 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 00:54:25.335109 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":212,"threads_started":1}
I20260924 00:54:25.336172 18002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:25.420672 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 17878
I20260924 00:54:25.434890 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 18010
I20260924 00:54:25.448383 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:25.435619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.222.3:44265)
0924 00:54:25.436176 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:25.436181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:25.436199 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:54:25.436253 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:25.436257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:25.436325 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:25.436415 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:25.437961 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:25.438734 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:25.439981 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:25.440407 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:25.440504 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:25.443109 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:25.443115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:25.443118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:25.443150 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:25.445951 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:25.446580 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:25.446584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:25.446586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:25.446714 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:25.447075 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:25.447079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:25.447081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:25.447387 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:25.447624 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:25.447887 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 00:54:25.448068 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":250,"threads_started":1}
I20260924 00:54:25.449075 18134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:25.533941 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 18010
2026-09-24T00:54:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64334 ms)
[----------] 30 tests from SecurityITest (302824 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:25 dist-test-slave-rtvz krb5kdc[18204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:25 dist-test-slave-rtvz krb5kdc[18204](info): set up 2 sockets
Sep 24 00:54:25 dist-test-slave-rtvz krb5kdc[18204](info): commencing operation
krb5kdc: starting...
W20260924 00:54:26.561298 888 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:26 dist-test-slave-rtvz krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:54:26.808471 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44059
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:44059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:27.143605 18218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:27.143920 18218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:27.144042 18218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:27.151542 18218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:27.151666 18218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:27.151712 18218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:27.151773 18218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:27.151818 18218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:27.161993 18218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:44059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:27.163374 18218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:27.164819 18218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:27.177143 18224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:27.177219 18223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:27.177242 18226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:27.177434 18218 server_base.cc:1034] running on GCE node
I20260924 00:54:27.178573 18218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:27.180001 18218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:27.181283 18218 hybrid_clock.cc:648] HybridClock initialized: now 1790211267181179 us; error 32 us; skew 500 ppm
I20260924 00:54:27.181660 18218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:27 dist-test-slave-rtvz krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:27.185029 18218 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:54:27.186846 18218 webserver.cc:460] Webserver started at http://127.0.222.62:39067/ using document root <none> and password file <none>
I20260924 00:54:27.187602 18218 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:27.187736 18218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:27.188048 18218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:27.191418 18218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "fe21d520dff040b3ad6fcd10f3850178"
format_stamp: "Formatted at 2026-09-24 00:54:27 on dist-test-slave-rtvz"
server_key: "e50911f0f51453d1f9aa0480f0f9f1a0"
server_key_iv: "b2a926b5dee3bc713851134d42abd2f6"
server_key_version: "encryptionkey@0"
I20260924 00:54:27.192287 18218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "fe21d520dff040b3ad6fcd10f3850178"
format_stamp: "Formatted at 2026-09-24 00:54:27 on dist-test-slave-rtvz"
server_key: "e50911f0f51453d1f9aa0480f0f9f1a0"
server_key_iv: "b2a926b5dee3bc713851134d42abd2f6"
server_key_version: "encryptionkey@0"
I20260924 00:54:27.198396 18218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 00:54:27.202239 18233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:27.203948 18218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:27.204164 18218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "fe21d520dff040b3ad6fcd10f3850178"
format_stamp: "Formatted at 2026-09-24 00:54:27 on dist-test-slave-rtvz"
server_key: "e50911f0f51453d1f9aa0480f0f9f1a0"
server_key_iv: "b2a926b5dee3bc713851134d42abd2f6"
server_key_version: "encryptionkey@0"
I20260924 00:54:27.204413 18218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:27.226032 18218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:27.231279 18218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:27.231606 18218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:27.250550 18218 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:44059
I20260924 00:54:27.250608 18285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:44059 every 8 connection(s)
I20260924 00:54:27.251946 18218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:27.254092 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 18218
I20260924 00:54:27.254318 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:27.255118 888 external_mini_cluster.cc:1428] Setting key cf233bdadf3e79fbd3802eaadad3db8a
I20260924 00:54:27.256616 18286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 00:54:27 dist-test-slave-rtvz krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:27.271036 18286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178: Bootstrap starting.
I20260924 00:54:27.274510 18286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:27.275668 18286 log.cc:824] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:27.279102 18286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178: No bootstrap required, opened a new log
I20260924 00:54:27.279662 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:27.257278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.0.222.62:44059)
0924 00:54:27.258043 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:27.258057 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:27.258098 (+ 41us) server_negotiation.cc:407] Connection header received
0924 00:54:27.259713 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:27.259737 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:27.260158 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:27.260509 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:27.262477 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:27.264007 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:27.265985 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:27.266751 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:27.266932 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:27.271897 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:27.271907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:27.271924 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:27.271973 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:27.275182 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:27.276086 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:27.276093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:27.276100 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:27.276344 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:27.276744 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:27.276750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:27.276765 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:27.277446 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:27.277671 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:27.278213 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 00:54:27.278615 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":356,"threads_started":1}
I20260924 00:54:27.286473 18286 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe21d520dff040b3ad6fcd10f3850178" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44059 } }
I20260924 00:54:27.286844 18286 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:27.286932 18286 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe21d520dff040b3ad6fcd10f3850178, State: Initialized, Role: FOLLOWER
I20260924 00:54:27.287500 18286 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe21d520dff040b3ad6fcd10f3850178" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44059 } }
I20260924 00:54:27.287717 18286 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:27.287840 18286 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:27.288002 18286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:27.290940 18286 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe21d520dff040b3ad6fcd10f3850178" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44059 } }
I20260924 00:54:27.291622 18286 leader_election.cc:302] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe21d520dff040b3ad6fcd10f3850178; no voters:
I20260924 00:54:27.292292 18286 leader_election.cc:288] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:27.292428 18291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:27.293037 18291 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [term 1 LEADER]: Becoming Leader. State: Replica: fe21d520dff040b3ad6fcd10f3850178, State: Running, Role: LEADER
I20260924 00:54:27.293653 18291 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe21d520dff040b3ad6fcd10f3850178" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44059 } }
I20260924 00:54:27.294375 18286 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:27.296592 18292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe21d520dff040b3ad6fcd10f3850178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe21d520dff040b3ad6fcd10f3850178" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44059 } } }
I20260924 00:54:27.296989 18292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:27.297297 18293 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe21d520dff040b3ad6fcd10f3850178. Latest consensus state: current_term: 1 leader_uuid: "fe21d520dff040b3ad6fcd10f3850178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe21d520dff040b3ad6fcd10f3850178" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 44059 } } }
I20260924 00:54:27.297712 18293 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:27.302696 18297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:27.307703 18297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:27.317798 18297 catalog_manager.cc:1347] Generated new cluster ID: 66f75ff8efc34f729c27b5ac7055d324
I20260924 00:54:27.318077 18297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:27.331985 18297 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:27.333920 18297 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:27.343070 18297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178: Generated new TSK 0
I20260924 00:54:27.344218 18297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:54:27.405246 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44059
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:27.734483 18312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:27.734764 18312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:27.734862 18312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:27.742287 18312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:27.742412 18312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:27.742527 18312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:54:27.753032 18312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:27.754377 18312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:27.755672 18312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:27.768702 18317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:27.768740 18320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:27.768766 18318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:27.769394 18312 server_base.cc:1034] running on GCE node
I20260924 00:54:27.769960 18312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:27.770924 18312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:27.772123 18312 hybrid_clock.cc:648] HybridClock initialized: now 1790211267772075 us; error 35 us; skew 500 ppm
I20260924 00:54:27.772532 18312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:27 dist-test-slave-rtvz krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:27.775714 18312 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:54:27.777318 18312 webserver.cc:460] Webserver started at http://127.0.222.1:38329/ using document root <none> and password file <none>
I20260924 00:54:27.778030 18312 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:27.778175 18312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:27.778463 18312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:27.781693 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "cb96e2d66db64b8ea4f802559b327861"
format_stamp: "Formatted at 2026-09-24 00:54:27 on dist-test-slave-rtvz"
server_key: "43b131a29d16db244e68559ae7ecaf51"
server_key_iv: "147ac391a523e74a97fe7693f1cfc056"
server_key_version: "encryptionkey@0"
I20260924 00:54:27.782464 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "cb96e2d66db64b8ea4f802559b327861"
format_stamp: "Formatted at 2026-09-24 00:54:27 on dist-test-slave-rtvz"
server_key: "43b131a29d16db244e68559ae7ecaf51"
server_key_iv: "147ac391a523e74a97fe7693f1cfc056"
server_key_version: "encryptionkey@0"
I20260924 00:54:27.787580 18312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 00:54:27.791067 18327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:27.792447 18312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:54:27.792586 18312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "cb96e2d66db64b8ea4f802559b327861"
format_stamp: "Formatted at 2026-09-24 00:54:27 on dist-test-slave-rtvz"
server_key: "43b131a29d16db244e68559ae7ecaf51"
server_key_iv: "147ac391a523e74a97fe7693f1cfc056"
server_key_version: "encryptionkey@0"
I20260924 00:54:27.792755 18312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:27.816144 18312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:27.820575 18312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:27.820848 18312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:27.821904 18312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:27.823236 18312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:27.823312 18312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:27.823405 18312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:27.823455 18312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:27.855046 18312 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:33315
I20260924 00:54:27.855129 18440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:33315 every 8 connection(s)
I20260924 00:54:27.856302 18312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:54:27.860057 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 18312
I20260924 00:54:27.860320 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:54:27.861163 888 external_mini_cluster.cc:1428] Setting key 699b1b88b73cf10e64427fb0cdc6857b
Sep 24 00:54:27 dist-test-slave-rtvz krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:27.878233 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:27.858794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:50695 (local address 127.0.222.62:44059)
0924 00:54:27.859216 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:27.859221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:27.860297 (+ 1076us) server_negotiation.cc:407] Connection header received
0924 00:54:27.862557 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:27.862563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:27.862645 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:27.862738 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:27.864696 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:27.865455 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:27.867519 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:27.867999 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:27.868102 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:27.872249 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:27.872253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:27.872257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:27.872292 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:27.874998 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:27.875815 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:27.875819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:27.875822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:27.875957 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:27.876385 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:27.876389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:27.876391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:27.876661 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:27.876797 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:27.877608 (+ 811us) server_negotiation.cc:299] Negotiation successful
0924 00:54:27.877862 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":219,"threads_started":1}
I20260924 00:54:27.879267 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:27.859241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44059 (local address 127.0.222.1:50695)
0924 00:54:27.860077 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:27.860128 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:27.862290 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:27.862994 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:27.863003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:27.863712 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:27.864500 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:27.864530 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:27.865611 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:27.865617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:27.867363 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:27.867378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:27.868220 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:27.868226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:27.868507 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:27.869257 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:27.869286 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:27.871971 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:27.875162 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:27.875169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:27.875184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:27.875687 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:27.876066 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:27.876070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:27.876074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:27.876274 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:27.876785 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:27.876794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:27.877157 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 00:54:27.877487 (+ 330us) client_negotiation.cc:240] Negotiation successful
0924 00:54:27.878052 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":300,"threads_started":1}
I20260924 00:54:27.880952 18441 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44059
I20260924 00:54:27.881281 18441 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:27.882033 18441 heartbeater.cc:511] Master 127.0.222.62:44059 requested a full tablet report, sending...
I20260924 00:54:27.884424 18250 ts_manager.cc:194] Registered new tserver with Master: cb96e2d66db64b8ea4f802559b327861 (127.0.222.1:33315)
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:54:27.886426 18250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:50695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:54:27.916453 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44059
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:28.248351 18447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:28.248661 18447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:28.248780 18447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:28.256295 18447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:28.256434 18447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:28.256588 18447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:54:28.266661 18447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:28.268045 18447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:28.269434 18447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:28.282243 18452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:28.282464 18453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:28.282729 18455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:28.283535 18447 server_base.cc:1034] running on GCE node
I20260924 00:54:28.284152 18447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:28.285177 18447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:28.286412 18447 hybrid_clock.cc:648] HybridClock initialized: now 1790211268286347 us; error 61 us; skew 500 ppm
I20260924 00:54:28.286726 18447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:28 dist-test-slave-rtvz krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:28.289961 18447 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:54:28.291512 18447 webserver.cc:460] Webserver started at http://127.0.222.2:34911/ using document root <none> and password file <none>
I20260924 00:54:28.292281 18447 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:28.292430 18447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:28.292784 18447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:28.296352 18447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "0333f33845444ea49efe793da4b32d61"
format_stamp: "Formatted at 2026-09-24 00:54:28 on dist-test-slave-rtvz"
server_key: "e375af95a54f6489330037b5c364e466"
server_key_iv: "89e2dc00155c41e7c8435840080895a1"
server_key_version: "encryptionkey@0"
I20260924 00:54:28.297170 18447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "0333f33845444ea49efe793da4b32d61"
format_stamp: "Formatted at 2026-09-24 00:54:28 on dist-test-slave-rtvz"
server_key: "e375af95a54f6489330037b5c364e466"
server_key_iv: "89e2dc00155c41e7c8435840080895a1"
server_key_version: "encryptionkey@0"
I20260924 00:54:28.302620 18447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:54:28.306314 18462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:28.307875 18447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:54:28.308084 18447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "0333f33845444ea49efe793da4b32d61"
format_stamp: "Formatted at 2026-09-24 00:54:28 on dist-test-slave-rtvz"
server_key: "e375af95a54f6489330037b5c364e466"
server_key_iv: "89e2dc00155c41e7c8435840080895a1"
server_key_version: "encryptionkey@0"
I20260924 00:54:28.308305 18447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:28.327726 18447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:28.332229 18447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:28.332512 18447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:28.333581 18447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:28.334972 18447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:28.335052 18447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:28.335196 18447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:28.335249 18447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:28.366217 18447 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:35891
I20260924 00:54:28.366256 18575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:35891 every 8 connection(s)
I20260924 00:54:28.367473 18447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:54:28.370702 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 18447
I20260924 00:54:28.370867 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:54:28.371282 888 external_mini_cluster.cc:1428] Setting key c95f85bf8f654ea3192a1d9fe94ece4c
Sep 24 00:54:28 dist-test-slave-rtvz krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:28.389300 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.369953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45649 (local address 127.0.222.62:44059)
0924 00:54:28.370154 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:28.370160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:28.371430 (+ 1270us) server_negotiation.cc:407] Connection header received
0924 00:54:28.373735 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:28.373740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:28.373827 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:28.373918 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:28.375909 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.376645 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.378751 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.379191 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.379287 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.383467 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:28.383474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:28.383478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:28.383517 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:28.386265 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:28.387003 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:28.387007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:28.387010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:28.387149 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:28.387563 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:28.387566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:28.387568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:28.387841 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:28.387970 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:28.388691 (+ 721us) server_negotiation.cc:299] Negotiation successful
0924 00:54:28.388931 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:54:28.390345 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.370485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44059 (local address 127.0.222.2:45649)
0924 00:54:28.371271 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:28.371316 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:28.373399 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:28.374202 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:28.374214 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:28.374934 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:28.375723 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.375748 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.376806 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.376810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.378609 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.378623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.379417 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.379423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.379708 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.380554 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:28.380590 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:28.383209 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:28.386412 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:28.386419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:28.386434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:28.386868 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:28.387256 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:28.387260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:28.387262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:28.387448 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:28.387971 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:28.387979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:28.388331 (+ 352us) client_negotiation.cc:769] Sending connection context
0924 00:54:28.388576 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 00:54:28.389140 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":316,"threads_started":1}
I20260924 00:54:28.391695 18576 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44059
I20260924 00:54:28.391990 18576 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:28.392774 18576 heartbeater.cc:511] Master 127.0.222.62:44059 requested a full tablet report, sending...
I20260924 00:54:28.394351 18250 ts_manager.cc:194] Registered new tserver with Master: 0333f33845444ea49efe793da4b32d61 (127.0.222.2:35891)
I20260924 00:54:28.395196 18250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:45649
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:54:28.429282 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44059
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:28.759876 18581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:28.760178 18581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:28.760317 18581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:28.767695 18581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:28.767830 18581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:28.768010 18581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:54:28.778342 18581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:44059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:28.779678 18581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:28.781041 18581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:28.794776 18587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:28.794706 18586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:28.794734 18589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:28.795418 18581 server_base.cc:1034] running on GCE node
I20260924 00:54:28.796217 18581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:28.797333 18581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:28.798560 18581 hybrid_clock.cc:648] HybridClock initialized: now 1790211268798496 us; error 56 us; skew 500 ppm
I20260924 00:54:28.798904 18581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:28 dist-test-slave-rtvz krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:28.802219 18581 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:54:28.803754 18581 webserver.cc:460] Webserver started at http://127.0.222.3:36987/ using document root <none> and password file <none>
I20260924 00:54:28.804452 18581 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:28.804569 18581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:28.804863 18581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:28.807940 18581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "2098ceb198fe4afc8a93d407456dac93"
format_stamp: "Formatted at 2026-09-24 00:54:28 on dist-test-slave-rtvz"
server_key: "8db22cbf15f78683538b8bb96fafcae9"
server_key_iv: "c2f03e1c88be69687a6dc9480e7c29ad"
server_key_version: "encryptionkey@0"
I20260924 00:54:28.808784 18581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "2098ceb198fe4afc8a93d407456dac93"
format_stamp: "Formatted at 2026-09-24 00:54:28 on dist-test-slave-rtvz"
server_key: "8db22cbf15f78683538b8bb96fafcae9"
server_key_iv: "c2f03e1c88be69687a6dc9480e7c29ad"
server_key_version: "encryptionkey@0"
I20260924 00:54:28.814137 18581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 00:54:28.817886 18596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:28.819341 18581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:28.819502 18581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "2098ceb198fe4afc8a93d407456dac93"
format_stamp: "Formatted at 2026-09-24 00:54:28 on dist-test-slave-rtvz"
server_key: "8db22cbf15f78683538b8bb96fafcae9"
server_key_iv: "c2f03e1c88be69687a6dc9480e7c29ad"
server_key_version: "encryptionkey@0"
I20260924 00:54:28.819702 18581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:28.844419 18581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:28.848734 18581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:28.849073 18581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:28.850147 18581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:28.851552 18581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:28.851635 18581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:28.851753 18581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:28.851804 18581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:28.883390 18581 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:41827
I20260924 00:54:28.883445 18709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:41827 every 8 connection(s)
I20260924 00:54:28.884742 18581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:54:28.889739 18441 heartbeater.cc:503] Master 127.0.222.62:44059 was elected leader, sending a full tablet report...
I20260924 00:54:28.893338 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 18581
I20260924 00:54:28.893543 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:54:28.894121 888 external_mini_cluster.cc:1428] Setting key a79806953fddaca979a1a1934585e0c3
Sep 24 00:54:28 dist-test-slave-rtvz krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:28.908715 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.887595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:53361 (local address 127.0.222.62:44059)
0924 00:54:28.887781 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:28.887786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:28.889150 (+ 1364us) server_negotiation.cc:407] Connection header received
0924 00:54:28.891358 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:28.891363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:28.891534 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:28.891621 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:28.893599 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.894869 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.897284 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.898030 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.898158 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.902392 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:28.902398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:28.902402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:28.902444 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:28.905160 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:28.906072 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:28.906076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:28.906079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:28.906225 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:28.906709 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:28.906712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:28.906715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:28.907029 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:28.907170 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:28.908030 (+ 860us) server_negotiation.cc:299] Negotiation successful
0924 00:54:28.908286 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:54:28.910121 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.888150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:44059 (local address 127.0.222.3:53361)
0924 00:54:28.888945 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:28.889000 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:28.890969 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:28.891978 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:28.891988 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:28.892653 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:28.893365 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.893388 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.895089 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.895094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.897047 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.897067 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.898301 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.898307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.898572 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.899278 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:28.899310 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:28.902099 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:28.905378 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:28.905389 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:28.905405 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:28.905909 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:28.906343 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:28.906347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:28.906350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:28.906578 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:28.907166 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:28.907176 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:28.907574 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 00:54:28.907871 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 00:54:28.908521 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":327,"threads_started":1}
I20260924 00:54:28.911448 18710 heartbeater.cc:347] Connected to a master server at 127.0.222.62:44059
I20260924 00:54:28.911743 18710 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:28.912556 18710 heartbeater.cc:511] Master 127.0.222.62:44059 requested a full tablet report, sending...
I20260924 00:54:28.914314 18250 ts_manager.cc:194] Registered new tserver with Master: 2098ceb198fe4afc8a93d407456dac93 (127.0.222.3:41827)
I20260924 00:54:28.915169 18250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:53361
I20260924 00:54:28.924703 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:54:28.946069 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.929826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.0.222.62:44059)
0924 00:54:28.930166 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:28.930173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:28.930439 (+ 266us) server_negotiation.cc:407] Connection header received
0924 00:54:28.930559 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:28.930564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:28.930635 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:28.930725 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:28.932391 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.933164 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.934950 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.935409 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.935570 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.938470 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:28.938477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:28.938481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:28.938528 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:28.942098 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:28.943093 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:28.943103 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:28.943106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:28.943385 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:28.944029 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:28.944037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:28.944039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:28.944495 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:28.944750 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:28.945296 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 00:54:28.945540 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260924 00:54:28.951392 18250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:54:28.958477 18250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:54:28.997581 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.973838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.0.222.1:33315)
0924 00:54:28.984735 (+ 10897us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:28.984749 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:28.986401 (+ 1652us) server_negotiation.cc:407] Connection header received
0924 00:54:28.986540 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:28.986550 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:28.986847 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:28.987102 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:28.987845 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.990771 (+ 2926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.994342 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.995607 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.995733 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.995988 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:28.996880 (+ 892us) server_negotiation.cc:299] Negotiation successful
0924 00:54:28.997085 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10625,"thread_start_us":10931,"threads_started":1}
I20260924 00:54:28.997576 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.974123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.0.222.3:41827)
0924 00:54:28.986683 (+ 12560us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:28.986692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:28.987407 (+ 715us) server_negotiation.cc:407] Connection header received
0924 00:54:28.987593 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:28.987602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:28.987875 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:28.988085 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:28.989279 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.991111 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.994401 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.995999 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.996177 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.996419 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:28.996928 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 00:54:28.997167 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12384,"thread_start_us":12220,"threads_started":1}
I20260924 00:54:28.998034 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.973721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41827 (local address 127.0.0.1:40940)
0924 00:54:28.986941 (+ 13220us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:28.986965 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:28.987108 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:28.988459 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:28.988465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:28.988497 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:28.988998 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.989011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.991300 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.991307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.994138 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.994155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.996247 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.996254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.996529 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.996604 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 00:54:28.996751 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 00:54:28.997651 (+ 900us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13107,"mutex_wait_us":684,"thread_start_us":11511,"threads_started":1}
I20260924 00:54:28.998862 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.973630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33315 (local address 127.0.0.1:52634)
0924 00:54:28.985752 (+ 12122us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:28.985813 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:28.986128 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:28.987108 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:28.987117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:28.987151 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:28.987627 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.987641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.990983 (+ 3342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.990989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.994138 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.994155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.995806 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.995812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.996625 (+ 813us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:28.996660 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:54:28.998240 (+ 1580us) client_negotiation.cc:240] Negotiation successful
0924 00:54:28.998393 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11778,"mutex_wait_us":488,"thread_start_us":13074,"threads_started":1}
I20260924 00:54:29.001894 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.987691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:35891 (local address 127.0.0.1:48478)
0924 00:54:28.992900 (+ 5209us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:28.992918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:28.993050 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:28.994128 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:28.994131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:28.994152 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:28.994508 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.994519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.996464 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.996469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:28.998755 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:28.998773 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.000836 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.000843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:29.001021 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.001056 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:54:29.001162 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 00:54:29.001490 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5066,"mutex_wait_us":5,"thread_start_us":4911,"threads_started":1}
I20260924 00:54:29.002046 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:28.987455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.0.222.2:35891)
0924 00:54:28.992386 (+ 4931us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:28.992396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:28.993184 (+ 788us) server_negotiation.cc:407] Connection header received
0924 00:54:28.993334 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:28.993342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:28.993653 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:28.993867 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:28.994677 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:28.996276 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:28.998945 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.000338 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.000469 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.000685 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:29.001506 (+ 821us) server_negotiation.cc:299] Negotiation successful
0924 00:54:29.001719 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4729,"thread_start_us":4594,"threads_started":1}
I20260924 00:54:29.002857 18645 tablet_service.cc:1467] Processing CreateTablet for tablet 804c9d2a72ed4b0bae7274777cdd538d (DEFAULT_TABLE table=test-table [id=fa5f25e94643496da7fbb9c167c248e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:29.002856 18375 tablet_service.cc:1467] Processing CreateTablet for tablet 804c9d2a72ed4b0bae7274777cdd538d (DEFAULT_TABLE table=test-table [id=fa5f25e94643496da7fbb9c167c248e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:29.004505 18645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804c9d2a72ed4b0bae7274777cdd538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:29.004519 18375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804c9d2a72ed4b0bae7274777cdd538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:29.005983 18510 tablet_service.cc:1467] Processing CreateTablet for tablet 804c9d2a72ed4b0bae7274777cdd538d (DEFAULT_TABLE table=test-table [id=fa5f25e94643496da7fbb9c167c248e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:29.007230 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804c9d2a72ed4b0bae7274777cdd538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:29.020107 18728 tablet_bootstrap.cc:492] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Bootstrap starting.
I20260924 00:54:29.024564 18729 tablet_bootstrap.cc:492] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Bootstrap starting.
I20260924 00:54:29.025540 18728 tablet_bootstrap.cc:654] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:29.026261 18730 tablet_bootstrap.cc:492] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Bootstrap starting.
I20260924 00:54:29.027316 18728 log.cc:824] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:29.028612 18729 tablet_bootstrap.cc:654] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:29.029716 18730 tablet_bootstrap.cc:654] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:29.030285 18729 log.cc:824] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:29.030826 18730 log.cc:824] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:29.031618 18728 tablet_bootstrap.cc:492] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: No bootstrap required, opened a new log
I20260924 00:54:29.032040 18728 ts_tablet_manager.cc:1397] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 00:54:29.034220 18730 tablet_bootstrap.cc:492] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: No bootstrap required, opened a new log
I20260924 00:54:29.034646 18730 ts_tablet_manager.cc:1397] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 00:54:29.040468 18729 tablet_bootstrap.cc:492] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: No bootstrap required, opened a new log
I20260924 00:54:29.040827 18729 ts_tablet_manager.cc:1397] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260924 00:54:29.042615 18730 raft_consensus.cc:348] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.043016 18730 raft_consensus.cc:374] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:29.043131 18730 raft_consensus.cc:729] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2098ceb198fe4afc8a93d407456dac93, State: Initialized, Role: FOLLOWER
I20260924 00:54:29.043336 18728 raft_consensus.cc:348] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.043802 18728 raft_consensus.cc:374] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:29.043735 18730 consensus_queue.cc:260] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.043901 18728 raft_consensus.cc:729] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0333f33845444ea49efe793da4b32d61, State: Initialized, Role: FOLLOWER
I20260924 00:54:29.044502 18728 consensus_queue.cc:260] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.050096 18710 heartbeater.cc:503] Master 127.0.222.62:44059 was elected leader, sending a full tablet report...
I20260924 00:54:29.050625 18576 heartbeater.cc:503] Master 127.0.222.62:44059 was elected leader, sending a full tablet report...
I20260924 00:54:29.051522 18730 ts_tablet_manager.cc:1428] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20260924 00:54:29.051692 18728 ts_tablet_manager.cc:1428] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Time spent starting tablet: real 0.019s user 0.016s sys 0.005s
I20260924 00:54:29.053385 18729 raft_consensus.cc:348] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.053864 18729 raft_consensus.cc:374] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:29.053970 18729 raft_consensus.cc:729] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb96e2d66db64b8ea4f802559b327861, State: Initialized, Role: FOLLOWER
I20260924 00:54:29.054728 18729 consensus_queue.cc:260] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.056623 18729 ts_tablet_manager.cc:1428] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
W20260924 00:54:29.108594 18442 tablet.cc:2369] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:54:29.119575 18577 tablet.cc:2369] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:54:29.136619 18711 tablet.cc:2369] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:54:29.143047 18735 raft_consensus.cc:482] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:29.143437 18735 raft_consensus.cc:504] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.145421 18735 leader_election.cc:288] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb96e2d66db64b8ea4f802559b327861 (127.0.222.1:33315), 2098ceb198fe4afc8a93d407456dac93 (127.0.222.3:41827)
I20260924 00:54:29.152812 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:29.145890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:46009 (local address 127.0.222.1:33315)
0924 00:54:29.146200 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:29.146206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:29.147018 (+ 812us) server_negotiation.cc:407] Connection header received
0924 00:54:29.147247 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:29.147253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:29.147333 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:29.147432 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:29.148106 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.148942 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.151039 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.151874 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.152018 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.152122 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:29.152419 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 00:54:29.152556 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260924 00:54:29.152997 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:29.145776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:33315 (local address 127.0.222.2:46009)
0924 00:54:29.146892 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:29.146911 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:29.147055 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:29.147479 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:29.147483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:29.147508 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:29.147937 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:29.147952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.149080 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.149083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:29.150880 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:29.150893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.152094 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.152100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:29.152274 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.152299 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 00:54:29.152391 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 00:54:29.152556 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":32,"thread_start_us":673,"threads_started":1}
I20260924 00:54:29.153085 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:29.145949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:50273 (local address 127.0.222.3:41827)
0924 00:54:29.146176 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:29.146184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:29.146870 (+ 686us) server_negotiation.cc:407] Connection header received
0924 00:54:29.147036 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:29.147042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:29.147130 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:29.147225 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:29.147993 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.148781 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.151115 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.152069 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.152200 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.152311 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:29.152662 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 00:54:29.152796 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:54:29.153296 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:29.145718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41827 (local address 127.0.222.2:50273)
0924 00:54:29.146743 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:29.146763 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:29.146891 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:29.147270 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:29.147274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:29.147297 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:29.147731 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:29.147742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.148929 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.148935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:29.150944 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:29.150965 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.152296 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.152303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:29.152499 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.152525 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 00:54:29.152618 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 00:54:29.152758 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":104,"thread_start_us":726,"threads_started":1}
I20260924 00:54:29.154430 18395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804c9d2a72ed4b0bae7274777cdd538d" candidate_uuid: "0333f33845444ea49efe793da4b32d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb96e2d66db64b8ea4f802559b327861" is_pre_election: true
I20260924 00:54:29.154430 18665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804c9d2a72ed4b0bae7274777cdd538d" candidate_uuid: "0333f33845444ea49efe793da4b32d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2098ceb198fe4afc8a93d407456dac93" is_pre_election: true
I20260924 00:54:29.155055 18395 raft_consensus.cc:2454] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0333f33845444ea49efe793da4b32d61 in term 0.
I20260924 00:54:29.155055 18665 raft_consensus.cc:2454] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0333f33845444ea49efe793da4b32d61 in term 0.
I20260924 00:54:29.155759 18465 leader_election.cc:302] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0333f33845444ea49efe793da4b32d61, 2098ceb198fe4afc8a93d407456dac93; no voters:
I20260924 00:54:29.156162 18735 raft_consensus.cc:2781] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:54:29.156339 18735 raft_consensus.cc:482] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:54:29.156450 18735 raft_consensus.cc:3037] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:29.158877 18735 raft_consensus.cc:504] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.159523 18735 leader_election.cc:288] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [CANDIDATE]: Term 1 election: Requested vote from peers cb96e2d66db64b8ea4f802559b327861 (127.0.222.1:33315), 2098ceb198fe4afc8a93d407456dac93 (127.0.222.3:41827)
I20260924 00:54:29.160010 18395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804c9d2a72ed4b0bae7274777cdd538d" candidate_uuid: "0333f33845444ea49efe793da4b32d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb96e2d66db64b8ea4f802559b327861"
I20260924 00:54:29.160130 18665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804c9d2a72ed4b0bae7274777cdd538d" candidate_uuid: "0333f33845444ea49efe793da4b32d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2098ceb198fe4afc8a93d407456dac93"
I20260924 00:54:29.160211 18395 raft_consensus.cc:3037] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:29.160338 18665 raft_consensus.cc:3037] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:29.162645 18665 raft_consensus.cc:2454] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0333f33845444ea49efe793da4b32d61 in term 1.
I20260924 00:54:29.162716 18395 raft_consensus.cc:2454] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0333f33845444ea49efe793da4b32d61 in term 1.
I20260924 00:54:29.163041 18465 leader_election.cc:302] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0333f33845444ea49efe793da4b32d61, 2098ceb198fe4afc8a93d407456dac93; no voters:
I20260924 00:54:29.163337 18735 raft_consensus.cc:2781] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:29.163833 18735 raft_consensus.cc:686] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 1 LEADER]: Becoming Leader. State: Replica: 0333f33845444ea49efe793da4b32d61, State: Running, Role: LEADER
I20260924 00:54:29.164381 18735 consensus_queue.cc:237] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } }
I20260924 00:54:29.169031 18250 catalog_manager.cc:5495] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0333f33845444ea49efe793da4b32d61 (127.0.222.2). New cstate: current_term: 1 leader_uuid: "0333f33845444ea49efe793da4b32d61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0333f33845444ea49efe793da4b32d61" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260924 00:54:29.200225 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:29.194807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.222.2:35891)
0924 00:54:29.195033 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:29.195039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:29.195118 (+ 79us) server_negotiation.cc:407] Connection header received
0924 00:54:29.195315 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:29.195319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:29.195458 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:29.195598 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:54:29.195916 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.196756 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.198261 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.198651 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.198737 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.199228 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:54:29.199713 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:54:29.199828 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:29.199908 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 00:54:29.199980 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 00:54:29.208719 18665 raft_consensus.cc:1261] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 1 FOLLOWER]: Refusing update from remote peer 0333f33845444ea49efe793da4b32d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:29.209309 18395 raft_consensus.cc:1261] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 1 FOLLOWER]: Refusing update from remote peer 0333f33845444ea49efe793da4b32d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:29.209843 18740 consensus_queue.cc:1035] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2098ceb198fe4afc8a93d407456dac93" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:29.210282 18735 consensus_queue.cc:1035] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb96e2d66db64b8ea4f802559b327861" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 33315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:29.229436 18745 mvcc.cc:205] Tried to move back new op lower bound from 7332705358668009472 to 7332705358501666816. Current Snapshot: MvccSnapshot[applied={T|T < 7332705358668009472}]
I20260924 00:54:29.238207 18250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:29.238461 18250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:29.242274 18250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe21d520dff040b3ad6fcd10f3850178: Sending DeleteTablet for 3 replicas of tablet 804c9d2a72ed4b0bae7274777cdd538d
I20260924 00:54:29.243490 18645 tablet_service.cc:1514] Processing DeleteTablet for tablet 804c9d2a72ed4b0bae7274777cdd538d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:29 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:40940
I20260924 00:54:29.243525 18375 tablet_service.cc:1514] Processing DeleteTablet for tablet 804c9d2a72ed4b0bae7274777cdd538d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:29 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:52634
I20260924 00:54:29.244398 18754 tablet_replica.cc:330] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: stopping tablet replica
I20260924 00:54:29.244580 18755 tablet_replica.cc:330] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: stopping tablet replica
I20260924 00:54:29.244892 18754 raft_consensus.cc:2229] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:29.244928 18755 raft_consensus.cc:2229] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:29.245139 18510 tablet_service.cc:1514] Processing DeleteTablet for tablet 804c9d2a72ed4b0bae7274777cdd538d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:29 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:48478
I20260924 00:54:29.245476 18754 raft_consensus.cc:2258] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:29.245479 18755 raft_consensus.cc:2258] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:29.247403 18756 tablet_replica.cc:330] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: stopping tablet replica
I20260924 00:54:29.247918 18756 raft_consensus.cc:2229] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:54:29.248265 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 18312
I20260924 00:54:29.248473 18754 ts_tablet_manager.cc:1905] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:29.248631 18756 raft_consensus.cc:2258] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:29.254910 18755 ts_tablet_manager.cc:1905] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:29.255777 18754 ts_tablet_manager.cc:1918] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:29.256038 18754 log.cc:1196] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/804c9d2a72ed4b0bae7274777cdd538d
I20260924 00:54:29.256657 18754 ts_tablet_manager.cc:1939] T 804c9d2a72ed4b0bae7274777cdd538d P 2098ceb198fe4afc8a93d407456dac93: Deleting consensus metadata
I20260924 00:54:29.258669 18236 catalog_manager.cc:4841] TS 2098ceb198fe4afc8a93d407456dac93 (127.0.222.3:41827): tablet 804c9d2a72ed4b0bae7274777cdd538d (table test-table [id=fa5f25e94643496da7fbb9c167c248e4]) successfully deleted
I20260924 00:54:29.263990 18755 ts_tablet_manager.cc:1918] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:29.264303 18755 log.cc:1196] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/804c9d2a72ed4b0bae7274777cdd538d
I20260924 00:54:29.264962 18755 ts_tablet_manager.cc:1939] T 804c9d2a72ed4b0bae7274777cdd538d P cb96e2d66db64b8ea4f802559b327861: Deleting consensus metadata
I20260924 00:54:29.266753 18756 ts_tablet_manager.cc:1905] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:29.266975 18235 catalog_manager.cc:4841] TS cb96e2d66db64b8ea4f802559b327861 (127.0.222.1:33315): tablet 804c9d2a72ed4b0bae7274777cdd538d (table test-table [id=fa5f25e94643496da7fbb9c167c248e4]) successfully deleted
I20260924 00:54:29.275005 18756 ts_tablet_manager.cc:1918] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:29.275309 18756 log.cc:1196] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/804c9d2a72ed4b0bae7274777cdd538d
I20260924 00:54:29.275977 18756 ts_tablet_manager.cc:1939] T 804c9d2a72ed4b0bae7274777cdd538d P 0333f33845444ea49efe793da4b32d61: Deleting consensus metadata
Sep 24 00:54:29 dist-test-slave-rtvz krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:54:29.278935 18235 catalog_manager.cc:4841] TS 0333f33845444ea49efe793da4b32d61 (127.0.222.2:35891): tablet 804c9d2a72ed4b0bae7274777cdd538d (table test-table [id=fa5f25e94643496da7fbb9c167c248e4]) successfully deleted
I20260924 00:54:29.285349 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:29.249620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.0.222.1:33315)
0924 00:54:29.249828 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:29.249835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:29.264708 (+ 14873us) server_negotiation.cc:407] Connection header received
0924 00:54:29.264971 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:29.264977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:29.265063 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:29.265181 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:29.269161 (+ 3980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.270296 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.272296 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.272920 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.273023 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.278442 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:29.278472 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:29.278493 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:29.278542 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:29.282267 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:29.283022 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:29.283031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:29.283035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:29.283200 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:29.283530 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:29.283535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:29.283538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:29.284149 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:29.284466 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:29.284678 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 00:54:29.284922 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:54:29.286108 18436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:29.377825 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 18312
I20260924 00:54:29.398977 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 18447
Sep 24 00:54:29 dist-test-slave-rtvz krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:54:29.414558 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:29.399706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.0.222.2:35891)
0924 00:54:29.399878 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:29.399884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:29.399998 (+ 114us) server_negotiation.cc:407] Connection header received
0924 00:54:29.400178 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:29.400182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:29.400270 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:29.400357 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:29.401663 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.402480 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.403788 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.404214 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.404348 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.408513 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:29.408528 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:29.408544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:29.408579 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:29.411648 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:29.412354 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:29.412362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:29.412364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:29.412515 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:29.412870 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:29.412874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:29.412876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:29.413396 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:29.413732 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:29.413997 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 00:54:29.414216 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:54:29.415256 18571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:29.529996 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 18447
I20260924 00:54:29.545820 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 18581
Sep 24 00:54:29 dist-test-slave-rtvz krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:54:29.560909 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:29.546657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.0.222.3:41827)
0924 00:54:29.546830 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:29.546836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:29.546856 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:54:29.547074 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:29.547078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:29.547144 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:29.547229 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:29.548504 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.549289 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.550547 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:29.550950 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:29.551071 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:29.555542 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:29.555558 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:29.555574 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:29.555616 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:29.558153 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:29.558855 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:29.558862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:29.558865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:29.558999 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:29.559396 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:29.559400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:29.559402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:29.559760 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:29.560048 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:29.560365 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 00:54:29.560578 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:54:29.561803 18705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:29.651757 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 18581
I20260924 00:54:29.665899 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 18218
I20260924 00:54:29.667066 18280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:29.777024 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 18218
2026-09-24T00:54:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4247 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T00:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:29Z Disabled control of system clock
I20260924 00:54:29.852509 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36441
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:36441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:54:30.187547 18765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:30.187866 18765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:30.187958 18765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:30.195333 18765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:30.195457 18765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:30.195502 18765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:30.195559 18765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:30.195601 18765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:30.205057 18765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:36441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:36441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:30.206405 18765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:30.208192 18765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:30.220400 18771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:30.220563 18773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:30.220404 18770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:30.221269 18765 server_base.cc:1034] running on GCE node
I20260924 00:54:30.222064 18765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:30.223529 18765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:30.224735 18765 hybrid_clock.cc:648] HybridClock initialized: now 1790211270224686 us; error 35 us; skew 500 ppm
I20260924 00:54:30.225087 18765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:30.233518 18765 webserver.cc:460] Webserver started at http://127.0.222.62:41869/ using document root <none> and password file <none>
I20260924 00:54:30.234297 18765 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:30.234396 18765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:30.234681 18765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:30.237802 18765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "30a7cadaa2244e2395994d21713f9721"
format_stamp: "Formatted at 2026-09-24 00:54:30 on dist-test-slave-rtvz"
server_key: "eda4a157eec8780246c85cad1764dd4d"
server_key_iv: "a5d042a20ba5cd3ef1b33777d1790606"
server_key_version: "encryptionkey@0"
I20260924 00:54:30.238619 18765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "30a7cadaa2244e2395994d21713f9721"
format_stamp: "Formatted at 2026-09-24 00:54:30 on dist-test-slave-rtvz"
server_key: "eda4a157eec8780246c85cad1764dd4d"
server_key_iv: "a5d042a20ba5cd3ef1b33777d1790606"
server_key_version: "encryptionkey@0"
I20260924 00:54:30.243856 18765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:54:30.247643 18779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:30.249312 18765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:30.249482 18765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "30a7cadaa2244e2395994d21713f9721"
format_stamp: "Formatted at 2026-09-24 00:54:30 on dist-test-slave-rtvz"
server_key: "eda4a157eec8780246c85cad1764dd4d"
server_key_iv: "a5d042a20ba5cd3ef1b33777d1790606"
server_key_version: "encryptionkey@0"
I20260924 00:54:30.249677 18765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:30.267670 18765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:30.268520 18765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:30.268759 18765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:30.286579 18765 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:36441
I20260924 00:54:30.286616 18831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:36441 every 8 connection(s)
I20260924 00:54:30.287958 18765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:30.291766 18832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:30.297387 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 18765
I20260924 00:54:30.297567 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:30.298080 888 external_mini_cluster.cc:1428] Setting key c78e8b7dc4e252286ce276873d4ef767
I20260924 00:54:30.304404 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721: Bootstrap starting.
I20260924 00:54:30.307741 18832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:30.308840 18832 log.cc:824] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:30.312311 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721: No bootstrap required, opened a new log
I20260924 00:54:30.316624 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:30.300344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.0.222.62:36441)
0924 00:54:30.304671 (+ 4327us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:30.304685 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:30.304736 (+ 51us) server_negotiation.cc:407] Connection header received
0924 00:54:30.306716 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:30.306744 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:30.307281 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:30.307694 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:30.308350 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:30.310087 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:30.312022 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:30.312790 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:30.312944 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:30.314193 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:30.314209 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:30.314224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:30.314783 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:30.314867 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:30.314888 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:30.314996 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:30.315216 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 00:54:30.315549 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3758,"thread_start_us":3448,"threads_started":1}
I20260924 00:54:30.320405 18832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7cadaa2244e2395994d21713f9721" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36441 } }
I20260924 00:54:30.320757 18832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:30.320848 18832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a7cadaa2244e2395994d21713f9721, State: Initialized, Role: FOLLOWER
I20260924 00:54:30.321448 18832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7cadaa2244e2395994d21713f9721" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36441 } }
I20260924 00:54:30.321671 18832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:30.321792 18832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:30.321933 18832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:30.324082 18832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7cadaa2244e2395994d21713f9721" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36441 } }
I20260924 00:54:30.324544 18832 leader_election.cc:302] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30a7cadaa2244e2395994d21713f9721; no voters:
I20260924 00:54:30.325218 18832 leader_election.cc:288] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:30.325424 18837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:30.326691 18837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [term 1 LEADER]: Becoming Leader. State: Replica: 30a7cadaa2244e2395994d21713f9721, State: Running, Role: LEADER
I20260924 00:54:30.327345 18832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:30.327304 18837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7cadaa2244e2395994d21713f9721" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36441 } }
I20260924 00:54:30.330155 18839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a7cadaa2244e2395994d21713f9721. Latest consensus state: current_term: 1 leader_uuid: "30a7cadaa2244e2395994d21713f9721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7cadaa2244e2395994d21713f9721" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36441 } } }
I20260924 00:54:30.330950 18838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30a7cadaa2244e2395994d21713f9721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7cadaa2244e2395994d21713f9721" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 36441 } } }
I20260924 00:54:30.330621 18839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:30.331260 18838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:30.333403 18847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:30.339007 18847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:30.348081 18847 catalog_manager.cc:1347] Generated new cluster ID: a30b973fb16246c0a7c2874c90f5da7c
I20260924 00:54:30.348225 18847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:30.376180 18847 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:30.378118 18847 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:30.404011 18847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721: Generated new TSK 0
I20260924 00:54:30.405126 18847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:30.411566 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36441
--builtin_ntp_servers=127.0.222.20:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:54:30.744774 18856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:30.745083 18856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:30.745175 18856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:30.752926 18856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:30.753070 18856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:30.753243 18856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:54:30.764024 18856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:30.765427 18856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:30.767242 18856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:30.780995 18862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:30.781167 18861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:30.781175 18864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:30.782198 18856 server_base.cc:1034] running on GCE node
I20260924 00:54:30.782862 18856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:30.783900 18856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:30.785113 18856 hybrid_clock.cc:648] HybridClock initialized: now 1790211270785063 us; error 40 us; skew 500 ppm
I20260924 00:54:30.785460 18856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:30.787945 18856 webserver.cc:460] Webserver started at http://127.0.222.1:34863/ using document root <none> and password file <none>
I20260924 00:54:30.788655 18856 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:30.788769 18856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:30.789105 18856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:30.792142 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "36028bb39b27408ca4d20453c5b440e5"
format_stamp: "Formatted at 2026-09-24 00:54:30 on dist-test-slave-rtvz"
server_key: "f12c06b87cbb78503b3d153c3cd0bdef"
server_key_iv: "d7a992168bbeae2471c62b4bad092655"
server_key_version: "encryptionkey@0"
I20260924 00:54:30.792944 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "36028bb39b27408ca4d20453c5b440e5"
format_stamp: "Formatted at 2026-09-24 00:54:30 on dist-test-slave-rtvz"
server_key: "f12c06b87cbb78503b3d153c3cd0bdef"
server_key_iv: "d7a992168bbeae2471c62b4bad092655"
server_key_version: "encryptionkey@0"
I20260924 00:54:30.798120 18856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 00:54:30.801796 18870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:30.803361 18856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:54:30.803532 18856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "36028bb39b27408ca4d20453c5b440e5"
format_stamp: "Formatted at 2026-09-24 00:54:30 on dist-test-slave-rtvz"
server_key: "f12c06b87cbb78503b3d153c3cd0bdef"
server_key_iv: "d7a992168bbeae2471c62b4bad092655"
server_key_version: "encryptionkey@0"
I20260924 00:54:30.803754 18856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:30.824103 18856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:30.825225 18856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:30.825541 18856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:30.826817 18856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:30.828408 18856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:30.828497 18856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:30.828572 18856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:30.828637 18856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:30.859875 18856 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:34283
I20260924 00:54:30.859941 18983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:34283 every 8 connection(s)
I20260924 00:54:30.861166 18856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:54:30.865650 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 18856
I20260924 00:54:30.865808 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:54:30.866232 888 external_mini_cluster.cc:1428] Setting key db062c925691527a11173f1616fa97c5
I20260924 00:54:30.869254 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36441
--builtin_ntp_servers=127.0.222.20:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:54:30.874210 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:30.863664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:54599 (local address 127.0.222.62:36441)
0924 00:54:30.864131 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:30.864135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:30.865383 (+ 1248us) server_negotiation.cc:407] Connection header received
0924 00:54:30.867309 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:30.867314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:30.867378 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:30.867469 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:30.868741 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:30.869813 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:30.871436 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:30.871946 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:30.872069 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:30.873399 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:30.873405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:30.873409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:30.873430 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:30.873481 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:30.873497 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:30.873598 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:30.873762 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 00:54:30.873862 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":362,"thread_start_us":206,"threads_started":1}
I20260924 00:54:30.875205 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:30.864249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36441 (local address 127.0.222.1:54599)
0924 00:54:30.865192 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:30.865266 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:30.867058 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:30.867727 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:30.867738 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:30.867857 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:30.868575 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:30.868593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:30.869957 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:30.869967 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:30.871295 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:30.871307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:30.872162 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:30.872167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:30.872385 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:30.873064 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:30.873067 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:30.873109 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:30.873220 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:30.873589 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:30.873596 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:30.873601 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:54:30.873739 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 00:54:30.874123 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":273,"threads_started":1}
I20260924 00:54:30.876808 18984 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36441
I20260924 00:54:30.877122 18984 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:30.877909 18984 heartbeater.cc:511] Master 127.0.222.62:36441 requested a full tablet report, sending...
I20260924 00:54:30.880028 18796 ts_manager.cc:194] Registered new tserver with Master: 36028bb39b27408ca4d20453c5b440e5 (127.0.222.1:34283)
I20260924 00:54:30.881603 18796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.1:54599
W20260924 00:54:31.210019 18988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:31.210343 18988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:31.210474 18988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:31.217867 18988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:31.217993 18988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:31.218142 18988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:54:31.227321 18988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:31.228700 18988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:31.230638 18988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:31.243980 18993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:31.243927 18996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:31.243919 18994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:31.244788 18988 server_base.cc:1034] running on GCE node
I20260924 00:54:31.245510 18988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:31.246572 18988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:31.247848 18988 hybrid_clock.cc:648] HybridClock initialized: now 1790211271247765 us; error 65 us; skew 500 ppm
I20260924 00:54:31.248250 18988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:31.256348 18988 webserver.cc:460] Webserver started at http://127.0.222.2:41157/ using document root <none> and password file <none>
I20260924 00:54:31.257181 18988 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:31.257370 18988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:31.257738 18988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:31.260874 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "696cad6b8aa24c6daaafaf2bef7559d1"
format_stamp: "Formatted at 2026-09-24 00:54:31 on dist-test-slave-rtvz"
server_key: "32fd33ffc619a2e16fdf7a2d5b48dd62"
server_key_iv: "5e9e8ff5bbbe3f052ec677fe643f56af"
server_key_version: "encryptionkey@0"
I20260924 00:54:31.261797 18988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "696cad6b8aa24c6daaafaf2bef7559d1"
format_stamp: "Formatted at 2026-09-24 00:54:31 on dist-test-slave-rtvz"
server_key: "32fd33ffc619a2e16fdf7a2d5b48dd62"
server_key_iv: "5e9e8ff5bbbe3f052ec677fe643f56af"
server_key_version: "encryptionkey@0"
I20260924 00:54:31.267621 18988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 00:54:31.272771 19002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:31.274771 18988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 00:54:31.274968 18988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "696cad6b8aa24c6daaafaf2bef7559d1"
format_stamp: "Formatted at 2026-09-24 00:54:31 on dist-test-slave-rtvz"
server_key: "32fd33ffc619a2e16fdf7a2d5b48dd62"
server_key_iv: "5e9e8ff5bbbe3f052ec677fe643f56af"
server_key_version: "encryptionkey@0"
I20260924 00:54:31.275243 18988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:31.298702 18988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:31.299643 18988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:31.299908 18988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:31.300947 18988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:31.302489 18988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:31.302587 18988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:31.302693 18988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:31.302771 18988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:31.337680 18988 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:42187
I20260924 00:54:31.337728 19117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:42187 every 8 connection(s)
I20260924 00:54:31.339048 18988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:54:31.344760 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 18988
I20260924 00:54:31.344928 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:54:31.345415 888 external_mini_cluster.cc:1428] Setting key 18d719d5ec3388cb45f550077162f748
I20260924 00:54:31.348548 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36441
--builtin_ntp_servers=127.0.222.20:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:54:31.352461 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.341553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:38395 (local address 127.0.222.62:36441)
0924 00:54:31.341787 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:31.341793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:31.343226 (+ 1433us) server_negotiation.cc:407] Connection header received
0924 00:54:31.344996 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:31.345002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:31.345079 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:31.345166 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:31.346468 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.347243 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.349642 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.350047 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.350140 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.351670 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:31.351675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:31.351678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:31.351696 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:31.351739 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:31.351750 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:31.351817 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:31.352098 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 00:54:31.352191 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 00:54:31.353829 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.342128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36441 (local address 127.0.222.2:38395)
0924 00:54:31.343060 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:31.343103 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:31.344709 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:31.345490 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:31.345500 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:31.345622 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:31.346303 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.346323 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.347399 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.347404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.349494 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.349509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.350258 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.350264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.350545 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.351316 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:31.351319 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:31.351375 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:31.351485 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:31.351832 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:31.351843 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:31.351852 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:54:31.352037 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 00:54:31.352455 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":302,"threads_started":1}
I20260924 00:54:31.355291 19118 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36441
I20260924 00:54:31.355589 19118 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:31.356463 19118 heartbeater.cc:511] Master 127.0.222.62:36441 requested a full tablet report, sending...
I20260924 00:54:31.357975 18796 ts_manager.cc:194] Registered new tserver with Master: 696cad6b8aa24c6daaafaf2bef7559d1 (127.0.222.2:42187)
I20260924 00:54:31.358779 18796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.2:38395
W20260924 00:54:31.680744 19121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:31.681080 19121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:31.681233 19121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:31.688657 19121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:31.688773 19121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:31.688887 19121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:54:31.698236 19121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:31.699535 19121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:31.701391 19121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:31.714751 19127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:31.714708 19126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:31.715025 19129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:31.715543 19121 server_base.cc:1034] running on GCE node
I20260924 00:54:31.716193 19121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:31.717180 19121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:31.718415 19121 hybrid_clock.cc:648] HybridClock initialized: now 1790211271718349 us; error 63 us; skew 500 ppm
I20260924 00:54:31.718739 19121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:31.721019 19121 webserver.cc:460] Webserver started at http://127.0.222.3:42271/ using document root <none> and password file <none>
I20260924 00:54:31.721761 19121 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:31.721870 19121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:31.722198 19121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:31.725267 19121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "d5b9bd07e4664e108381ee14922cc7d2"
format_stamp: "Formatted at 2026-09-24 00:54:31 on dist-test-slave-rtvz"
server_key: "82b887eb934b61b562ec0047cf11c0c0"
server_key_iv: "1c3836ec835fd4f73460aad0808725e0"
server_key_version: "encryptionkey@0"
I20260924 00:54:31.726099 19121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "d5b9bd07e4664e108381ee14922cc7d2"
format_stamp: "Formatted at 2026-09-24 00:54:31 on dist-test-slave-rtvz"
server_key: "82b887eb934b61b562ec0047cf11c0c0"
server_key_iv: "1c3836ec835fd4f73460aad0808725e0"
server_key_version: "encryptionkey@0"
I20260924 00:54:31.731640 19121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:54:31.735440 19136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:31.736901 19121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:54:31.737123 19121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "d5b9bd07e4664e108381ee14922cc7d2"
format_stamp: "Formatted at 2026-09-24 00:54:31 on dist-test-slave-rtvz"
server_key: "82b887eb934b61b562ec0047cf11c0c0"
server_key_iv: "1c3836ec835fd4f73460aad0808725e0"
server_key_version: "encryptionkey@0"
I20260924 00:54:31.737390 19121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:31.778803 19121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:31.779757 19121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:31.780032 19121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:31.781255 19121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:31.782783 19121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:31.782886 19121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:31.782992 19121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:31.783075 19121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:31.813562 19121 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:39821
I20260924 00:54:31.813630 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:39821 every 8 connection(s)
I20260924 00:54:31.814889 19121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:54:31.816264 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 19121
I20260924 00:54:31.816438 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:54:31.816998 888 external_mini_cluster.cc:1428] Setting key a892adc1b9614b9f48c62a6de53beaea
I20260924 00:54:31.827456 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.817539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:60875 (local address 127.0.222.62:36441)
0924 00:54:31.817977 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:31.817982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:31.819427 (+ 1445us) server_negotiation.cc:407] Connection header received
0924 00:54:31.821321 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:31.821326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:31.821404 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:31.821493 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:31.822530 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.823251 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.824754 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.825180 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.825319 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.826591 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:31.826595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:31.826599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:31.826616 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:31.826649 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:31.826660 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:31.826732 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:31.826947 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 00:54:31.827085 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260924 00:54:31.828439 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.818261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:36441 (local address 127.0.222.3:60875)
0924 00:54:31.819287 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:31.819329 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:31.821051 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:31.821720 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:31.821728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:31.821820 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:31.822371 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.822388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.823391 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.823395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.824620 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.824633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.825415 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.825422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.825639 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.826282 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:31.826285 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:31.826325 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:31.826407 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:31.826744 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:31.826755 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:31.826763 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 00:54:31.826921 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 00:54:31.827334 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":291,"threads_started":1}
I20260924 00:54:31.829854 19250 heartbeater.cc:347] Connected to a master server at 127.0.222.62:36441
I20260924 00:54:31.830219 19250 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:31.831142 19250 heartbeater.cc:511] Master 127.0.222.62:36441 requested a full tablet report, sending...
I20260924 00:54:31.832490 18796 ts_manager.cc:194] Registered new tserver with Master: d5b9bd07e4664e108381ee14922cc7d2 (127.0.222.3:39821)
I20260924 00:54:31.833624 18796 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.3:60875
I20260924 00:54:31.844400 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:54:31.854667 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.848324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.0.222.62:36441)
0924 00:54:31.848568 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:31.848575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:31.848966 (+ 391us) server_negotiation.cc:407] Connection header received
0924 00:54:31.849059 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:31.849063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:31.849150 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:31.849308 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:31.849820 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.850650 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.852164 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.852618 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.852720 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.853875 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:31.853881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:31.853885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:31.853979 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:31.854013 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:31.854027 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:31.854102 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:31.854247 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 00:54:31.854353 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:54:31.858692 18796 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:54:31.865610 18796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:54:31.884618 18984 heartbeater.cc:503] Master 127.0.222.62:36441 was elected leader, sending a full tablet report...
I20260924 00:54:31.903484 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.878293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.0.222.2:42187)
0924 00:54:31.894136 (+ 15843us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:31.894145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:31.894171 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:54:31.894254 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:31.894260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:31.894457 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:31.894630 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:31.896245 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.897746 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.900416 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.902195 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.902410 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.902651 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:31.902903 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 00:54:31.903095 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15678,"thread_start_us":15539,"threads_started":1}
I20260924 00:54:31.903493 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.878036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:42187 (local address 127.0.0.1:51974)
0924 00:54:31.892430 (+ 14394us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:31.892502 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:31.892728 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:31.894870 (+ 2142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:31.894879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:31.894928 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:31.896051 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.896063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.897906 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.897912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.900199 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.900212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.902408 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.902414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.902622 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.902675 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:54:31.902820 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 00:54:31.903027 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14017,"mutex_wait_us":762,"spinlock_wait_cycles":6144,"thread_start_us":14924,"threads_started":1}
I20260924 00:54:31.904114 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.894260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:39821 (local address 127.0.0.1:52966)
0924 00:54:31.895394 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:31.895415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:31.895527 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:31.897618 (+ 2091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:31.897623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:31.897650 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:31.898082 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.898092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.899977 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.899983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.901520 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.901532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.903251 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.903257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.903451 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.903486 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:54:31.903565 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 00:54:31.903687 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":3,"thread_start_us":843,"threads_started":1}
I20260924 00:54:31.904543 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.878186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.0.1:47240)
0924 00:54:31.893476 (+ 15290us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:31.893495 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:31.893650 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:31.894870 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:31.894879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:31.894921 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:31.895477 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.895491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.897491 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.897497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.900346 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:31.900359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.903767 (+ 3408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.903774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:31.903953 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.903983 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 00:54:31.904065 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:54:31.904197 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15133,"mutex_wait_us":132,"spinlock_wait_cycles":1408,"thread_start_us":15545,"threads_started":1}
I20260924 00:54:31.905222 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.878660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.0.222.1:34283)
0924 00:54:31.893497 (+ 14837us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:31.893506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:31.893929 (+ 423us) server_negotiation.cc:407] Connection header received
0924 00:54:31.894051 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:31.894059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:31.894322 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:31.894553 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:31.895688 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.897333 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.900552 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.902195 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.902362 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.902677 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:31.904460 (+ 1783us) server_negotiation.cc:299] Negotiation successful
0924 00:54:31.904645 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14658,"thread_start_us":14839,"threads_started":1}
I20260924 00:54:31.906309 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:31.894309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.0.222.3:39821)
0924 00:54:31.897050 (+ 2741us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:31.897056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:31.897085 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:54:31.897166 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:31.897172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:31.897412 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:31.897618 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:31.898270 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.899799 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.901686 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:31.902762 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:31.902874 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:31.903128 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:31.904414 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0924 00:54:31.904588 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2581,"thread_start_us":2438,"threads_started":1}
I20260924 00:54:31.908100 19052 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0cda8da09b4e469cdcddc16a399636 (DEFAULT_TABLE table=test-table [id=8afff3e3fb7d48db98491e76556af24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:31.908991 18918 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0cda8da09b4e469cdcddc16a399636 (DEFAULT_TABLE table=test-table [id=8afff3e3fb7d48db98491e76556af24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:31.909767 19052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0cda8da09b4e469cdcddc16a399636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:31.909996 19184 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0cda8da09b4e469cdcddc16a399636 (DEFAULT_TABLE table=test-table [id=8afff3e3fb7d48db98491e76556af24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:31.910547 18918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0cda8da09b4e469cdcddc16a399636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:31.911369 19184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0cda8da09b4e469cdcddc16a399636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:31.921403 19268 tablet_bootstrap.cc:492] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Bootstrap starting.
I20260924 00:54:31.924808 19268 tablet_bootstrap.cc:654] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:31.926380 19268 log.cc:824] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:31.927027 19269 tablet_bootstrap.cc:492] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5: Bootstrap starting.
I20260924 00:54:31.929603 19270 tablet_bootstrap.cc:492] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Bootstrap starting.
I20260924 00:54:31.931131 19269 tablet_bootstrap.cc:654] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:31.931417 19268 tablet_bootstrap.cc:492] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: No bootstrap required, opened a new log
I20260924 00:54:31.931681 19268 ts_tablet_manager.cc:1397] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 00:54:31.932497 19269 log.cc:824] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:31.933621 19270 tablet_bootstrap.cc:654] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:31.935063 19270 log.cc:824] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:31.937789 19269 tablet_bootstrap.cc:492] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5: No bootstrap required, opened a new log
I20260924 00:54:31.938067 19269 ts_tablet_manager.cc:1397] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20260924 00:54:31.938086 19270 tablet_bootstrap.cc:492] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: No bootstrap required, opened a new log
I20260924 00:54:31.938366 19270 ts_tablet_manager.cc:1397] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 00:54:31.940945 19268 raft_consensus.cc:348] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:31.941569 19268 raft_consensus.cc:374] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:31.941687 19268 raft_consensus.cc:729] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696cad6b8aa24c6daaafaf2bef7559d1, State: Initialized, Role: FOLLOWER
I20260924 00:54:31.942471 19268 consensus_queue.cc:260] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:31.943879 19118 heartbeater.cc:503] Master 127.0.222.62:36441 was elected leader, sending a full tablet report...
I20260924 00:54:31.944569 19268 ts_tablet_manager.cc:1428] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 00:54:31.947242 19269 raft_consensus.cc:348] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:31.947754 19269 raft_consensus.cc:374] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:31.947892 19269 raft_consensus.cc:729] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36028bb39b27408ca4d20453c5b440e5, State: Initialized, Role: FOLLOWER
I20260924 00:54:31.947661 19270 raft_consensus.cc:348] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:31.948108 19270 raft_consensus.cc:374] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:31.948192 19270 raft_consensus.cc:729] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b9bd07e4664e108381ee14922cc7d2, State: Initialized, Role: FOLLOWER
I20260924 00:54:31.948568 19269 consensus_queue.cc:260] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:31.948776 19270 consensus_queue.cc:260] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:31.950573 19269 ts_tablet_manager.cc:1428] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260924 00:54:31.951704 19250 heartbeater.cc:503] Master 127.0.222.62:36441 was elected leader, sending a full tablet report...
I20260924 00:54:31.952193 19270 ts_tablet_manager.cc:1428] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260924 00:54:32.022954 19274 raft_consensus.cc:482] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:32.023288 19274 raft_consensus.cc:504] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:32.025182 19274 leader_election.cc:288] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36028bb39b27408ca4d20453c5b440e5 (127.0.222.1:34283), d5b9bd07e4664e108381ee14922cc7d2 (127.0.222.3:39821)
I20260924 00:54:32.032404 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.025777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:51759 (local address 127.0.222.1:34283)
0924 00:54:32.025968 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:32.025973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:32.026508 (+ 535us) server_negotiation.cc:407] Connection header received
0924 00:54:32.026702 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:32.026707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:32.026803 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:32.026890 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:32.027612 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.028514 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.030592 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.031518 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.031630 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:32.031731 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:32.032033 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 00:54:32.032160 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 00:54:32.032513 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.025648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.222.2:51759)
0924 00:54:32.026386 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.026402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:32.026554 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:32.027052 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:32.027057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:32.027087 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:32.027460 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:32.027469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.028649 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.028654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:32.030418 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:32.030430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.031697 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.031703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:32.031878 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:32.031902 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 00:54:32.031987 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:54:32.032119 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":7,"thread_start_us":440,"threads_started":1}
I20260924 00:54:32.032511 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.025751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:36685 (local address 127.0.222.3:39821)
0924 00:54:32.025947 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:32.025953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:32.027193 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 00:54:32.027388 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:32.027393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:32.027475 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:32.027588 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:32.028146 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.028962 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.030870 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.031641 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.031760 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:32.031857 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:32.032145 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 00:54:32.032278 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:54:32.032600 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.025662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:39821 (local address 127.0.222.2:36685)
0924 00:54:32.027087 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.027104 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:32.027234 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:32.027598 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:32.027602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:32.027627 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:32.027971 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:32.027988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.029114 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.029120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:32.030649 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:32.030662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.031832 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.031838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:32.031998 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:32.032020 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 00:54:32.032099 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 00:54:32.032218 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"mutex_wait_us":50,"thread_start_us":1071,"threads_started":1}
I20260924 00:54:32.033655 18939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0cda8da09b4e469cdcddc16a399636" candidate_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36028bb39b27408ca4d20453c5b440e5" is_pre_election: true
I20260924 00:54:32.033807 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0cda8da09b4e469cdcddc16a399636" candidate_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b9bd07e4664e108381ee14922cc7d2" is_pre_election: true
I20260924 00:54:32.034189 18939 raft_consensus.cc:2454] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696cad6b8aa24c6daaafaf2bef7559d1 in term 0.
I20260924 00:54:32.034296 19204 raft_consensus.cc:2454] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 696cad6b8aa24c6daaafaf2bef7559d1 in term 0.
I20260924 00:54:32.034823 19006 leader_election.cc:302] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36028bb39b27408ca4d20453c5b440e5, 696cad6b8aa24c6daaafaf2bef7559d1; no voters:
I20260924 00:54:32.035207 19274 raft_consensus.cc:2781] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:54:32.035333 19274 raft_consensus.cc:482] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:54:32.035427 19274 raft_consensus.cc:3037] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:32.037721 19274 raft_consensus.cc:504] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:32.038441 19274 leader_election.cc:288] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [CANDIDATE]: Term 1 election: Requested vote from peers 36028bb39b27408ca4d20453c5b440e5 (127.0.222.1:34283), d5b9bd07e4664e108381ee14922cc7d2 (127.0.222.3:39821)
I20260924 00:54:32.038986 19204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0cda8da09b4e469cdcddc16a399636" candidate_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b9bd07e4664e108381ee14922cc7d2"
I20260924 00:54:32.039054 18939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0cda8da09b4e469cdcddc16a399636" candidate_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36028bb39b27408ca4d20453c5b440e5"
I20260924 00:54:32.039176 19204 raft_consensus.cc:3037] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:32.039309 18939 raft_consensus.cc:3037] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:32.041481 19204 raft_consensus.cc:2454] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696cad6b8aa24c6daaafaf2bef7559d1 in term 1.
I20260924 00:54:32.041800 18939 raft_consensus.cc:2454] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 696cad6b8aa24c6daaafaf2bef7559d1 in term 1.
I20260924 00:54:32.041899 19003 leader_election.cc:302] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696cad6b8aa24c6daaafaf2bef7559d1, d5b9bd07e4664e108381ee14922cc7d2; no voters:
I20260924 00:54:32.042268 19274 raft_consensus.cc:2781] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:32.042711 19274 raft_consensus.cc:686] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 1 LEADER]: Becoming Leader. State: Replica: 696cad6b8aa24c6daaafaf2bef7559d1, State: Running, Role: LEADER
I20260924 00:54:32.043179 19274 consensus_queue.cc:237] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } }
I20260924 00:54:32.047787 18795 catalog_manager.cc:5495] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 696cad6b8aa24c6daaafaf2bef7559d1 (127.0.222.2). New cstate: current_term: 1 leader_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "696cad6b8aa24c6daaafaf2bef7559d1" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 } health_report { overall_health: UNKNOWN } } }
W20260924 00:54:32.066579 19251 tablet.cc:2369] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:54:32.070580 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.064726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.0.222.2:42187)
0924 00:54:32.064910 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:32.064917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:32.064939 (+ 22us) server_negotiation.cc:407] Connection header received
0924 00:54:32.065179 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:32.065183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:32.065334 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:32.065478 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:54:32.065797 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.066549 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.068315 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.068772 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.068879 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:32.069503 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:54:32.070050 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:54:32.070164 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:32.070255 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 00:54:32.070325 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:54:32.079025 18939 raft_consensus.cc:1261] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 1 FOLLOWER]: Refusing update from remote peer 696cad6b8aa24c6daaafaf2bef7559d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:32.079025 19204 raft_consensus.cc:1261] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 1 FOLLOWER]: Refusing update from remote peer 696cad6b8aa24c6daaafaf2bef7559d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:32.080044 19279 consensus_queue.cc:1035] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36028bb39b27408ca4d20453c5b440e5" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 34283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:32.080430 19274 consensus_queue.cc:1035] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b9bd07e4664e108381ee14922cc7d2" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:54:32.091034 19119 tablet.cc:2369] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:54:32.091859 19282 mvcc.cc:205] Tried to move back new op lower bound from 7332705370425470976 to 7332705370292625408. Current Snapshot: MvccSnapshot[applied={T|T < 7332705370425470976}]
I20260924 00:54:32.106460 18796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:32.106729 18796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:32.110392 18796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30a7cadaa2244e2395994d21713f9721: Sending DeleteTablet for 3 replicas of tablet 8b0cda8da09b4e469cdcddc16a399636
I20260924 00:54:32.111711 18918 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b0cda8da09b4e469cdcddc16a399636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:32 UTC) from {username='slave'} at 127.0.0.1:47240
I20260924 00:54:32.112046 19052 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b0cda8da09b4e469cdcddc16a399636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:32 UTC) from {username='slave'} at 127.0.0.1:51974
I20260924 00:54:32.112080 19184 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b0cda8da09b4e469cdcddc16a399636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:32 UTC) from {username='slave'} at 127.0.0.1:52966
I20260924 00:54:32.113473 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 18856
I20260924 00:54:32.124029 19292 tablet_replica.cc:330] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5: stopping tablet replica
I20260924 00:54:32.124641 19292 raft_consensus.cc:2229] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:32.125279 19292 raft_consensus.cc:2258] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:32.126302 19293 tablet_replica.cc:330] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: stopping tablet replica
I20260924 00:54:32.126832 19293 raft_consensus.cc:2229] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:54:32.127609 19293 raft_consensus.cc:2258] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:32.130368 19292 ts_tablet_manager.cc:1905] T 8b0cda8da09b4e469cdcddc16a399636 P 36028bb39b27408ca4d20453c5b440e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:32.133222 19293 ts_tablet_manager.cc:1905] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:32.139144 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.114773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.0.222.1:34283)
0924 00:54:32.114948 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:32.114954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:32.130304 (+ 15350us) server_negotiation.cc:407] Connection header received
0924 00:54:32.130688 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:32.130695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:32.130812 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:32.130891 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:32.131703 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.132785 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.134954 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.135602 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.135706 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:32.137311 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:32.137325 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:32.137344 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:32.138155 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:32.138221 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:32.138247 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:32.138368 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:32.138626 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 00:54:32.138755 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:54:32.140553 18979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:32.141875 19294 tablet_replica.cc:330] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: stopping tablet replica
I20260924 00:54:32.142288 19294 raft_consensus.cc:2229] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:32.142784 19294 raft_consensus.cc:2258] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:32.143697 19293 ts_tablet_manager.cc:1918] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:32.143889 19293 log.cc:1196] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/8b0cda8da09b4e469cdcddc16a399636
I20260924 00:54:32.144438 19293 ts_tablet_manager.cc:1939] T 8b0cda8da09b4e469cdcddc16a399636 P 696cad6b8aa24c6daaafaf2bef7559d1: Deleting consensus metadata
I20260924 00:54:32.145092 19294 ts_tablet_manager.cc:1905] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:32.146320 18780 catalog_manager.cc:4841] TS 696cad6b8aa24c6daaafaf2bef7559d1 (127.0.222.2:42187): tablet 8b0cda8da09b4e469cdcddc16a399636 (table test-table [id=8afff3e3fb7d48db98491e76556af24e]) successfully deleted
I20260924 00:54:32.152029 19294 ts_tablet_manager.cc:1918] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:32.152225 19294 log.cc:1196] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/8b0cda8da09b4e469cdcddc16a399636
I20260924 00:54:32.152745 19294 ts_tablet_manager.cc:1939] T 8b0cda8da09b4e469cdcddc16a399636 P d5b9bd07e4664e108381ee14922cc7d2: Deleting consensus metadata
I20260924 00:54:32.154690 18780 catalog_manager.cc:4841] TS d5b9bd07e4664e108381ee14922cc7d2 (127.0.222.3:39821): tablet 8b0cda8da09b4e469cdcddc16a399636 (table test-table [id=8afff3e3fb7d48db98491e76556af24e]) successfully deleted
I20260924 00:54:32.236586 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 18856
I20260924 00:54:32.251088 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 18988
W20260924 00:54:32.251186 18783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:34283 (error 108)
I20260924 00:54:32.253342 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.252771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.0.1:47248)
0924 00:54:32.253031 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.253139 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260924 00:54:32.253623 18783 catalog_manager.cc:4584] TS 36028bb39b27408ca4d20453c5b440e5 (127.0.222.1:34283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b0cda8da09b4e469cdcddc16a399636: Network error: Client connection negotiation failed: client connection to 127.0.222.1:34283: connect: Connection refused (error 111)
I20260924 00:54:32.259526 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.252081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.0.222.2:42187)
0924 00:54:32.252261 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:32.252268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:32.252339 (+ 71us) server_negotiation.cc:407] Connection header received
0924 00:54:32.252572 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:32.252578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:32.252693 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:32.252805 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:32.253301 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.254282 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.256189 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.256737 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.256878 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:32.258121 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:32.258131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:32.258145 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:32.258785 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:32.258836 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:32.258856 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:32.258941 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:32.259084 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 00:54:32.259171 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:54:32.260370 19113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:32.307263 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.306716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.0.1:47254)
0924 00:54:32.306963 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.307073 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 00:54:32.308352 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.307951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.0.1:47268)
0924 00:54:32.308120 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.308218 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 00:54:32.369042 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.368415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.0.1:47282)
0924 00:54:32.368680 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.368812 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 00:54:32.370172 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.369768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.0.1:47292)
0924 00:54:32.369942 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.370032 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 00:54:32.376205 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 18988
I20260924 00:54:32.392246 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 19121
I20260924 00:54:32.399192 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.393135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.0.222.3:39821)
0924 00:54:32.393365 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:32.393370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:32.393416 (+ 46us) server_negotiation.cc:407] Connection header received
0924 00:54:32.393568 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:32.393583 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:32.393671 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:32.393756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:32.394119 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.394859 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.396101 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:32.396517 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:32.396616 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:32.397666 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:32.397677 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:32.397692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:32.398378 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:32.398430 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:32.398454 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:32.398554 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:32.398682 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 00:54:32.398781 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260924 00:54:32.399914 19245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:32.451399 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.450827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.0.1:47304)
0924 00:54:32.451094 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.451210 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 00:54:32.452598 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:32.452152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:34283 (local address 127.0.0.1:47310)
0924 00:54:32.452371 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:32.452466 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:34283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 00:54:32.488703 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 19121
I20260924 00:54:32.503022 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 18765
I20260924 00:54:32.504153 18826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:32.611446 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 18765
2026-09-24T00:54:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2830 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:32 dist-test-slave-rtvz krb5kdc[19303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:32 dist-test-slave-rtvz krb5kdc[19303](info): set up 2 sockets
Sep 24 00:54:32 dist-test-slave-rtvz krb5kdc[19303](info): commencing operation
krb5kdc: starting...
W20260924 00:54:33.632443 888 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:33 dist-test-slave-rtvz krb5kdc[19303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:54:33.877477 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45309
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:34.213991 19316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:34.214290 19316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:34.214416 19316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:34.221787 19316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:34.221903 19316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:34.221978 19316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:34.222059 19316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:34.222115 19316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:34.232028 19316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:34.233402 19316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:34.234736 19316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:34.246340 19321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:34.246479 19322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:34.246558 19324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:34.247328 19316 server_base.cc:1034] running on GCE node
I20260924 00:54:34.248124 19316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:34.249611 19316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:34.250818 19316 hybrid_clock.cc:648] HybridClock initialized: now 1790211274250766 us; error 32 us; skew 500 ppm
I20260924 00:54:34.251216 19316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:34 dist-test-slave-rtvz krb5kdc[19303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:34.254436 19316 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:54:34.256120 19316 webserver.cc:460] Webserver started at http://127.0.222.62:43315/ using document root <none> and password file <none>
I20260924 00:54:34.256796 19316 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:34.256911 19316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:34.257232 19316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:34.260192 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "6e268331cf19483fb460a7e6f3910645"
format_stamp: "Formatted at 2026-09-24 00:54:34 on dist-test-slave-rtvz"
server_key: "2cf03b8869980d445d11237eb3acec06"
server_key_iv: "510e646dda6ac6e9f5149367ae8d8252"
server_key_version: "encryptionkey@0"
I20260924 00:54:34.260931 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "6e268331cf19483fb460a7e6f3910645"
format_stamp: "Formatted at 2026-09-24 00:54:34 on dist-test-slave-rtvz"
server_key: "2cf03b8869980d445d11237eb3acec06"
server_key_iv: "510e646dda6ac6e9f5149367ae8d8252"
server_key_version: "encryptionkey@0"
I20260924 00:54:34.266224 19316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:54:34.269877 19331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:34.271332 19316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:34.271517 19316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "6e268331cf19483fb460a7e6f3910645"
format_stamp: "Formatted at 2026-09-24 00:54:34 on dist-test-slave-rtvz"
server_key: "2cf03b8869980d445d11237eb3acec06"
server_key_iv: "510e646dda6ac6e9f5149367ae8d8252"
server_key_version: "encryptionkey@0"
I20260924 00:54:34.271709 19316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:34.286706 19316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:34.291167 19316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:34.291464 19316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:34.309739 19316 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45309
I20260924 00:54:34.309808 19383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45309 every 8 connection(s)
I20260924 00:54:34.311185 19316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:34.312558 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 19316
I20260924 00:54:34.312743 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:34.313402 888 external_mini_cluster.cc:1428] Setting key 06da11a243b2276e773b09549986c62c
I20260924 00:54:34.315852 19384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 00:54:34 dist-test-slave-rtvz krb5kdc[19303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:34.327960 19384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645: Bootstrap starting.
I20260924 00:54:34.331372 19384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:34.332419 19384 log.cc:824] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:34.335392 19384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645: No bootstrap required, opened a new log
I20260924 00:54:34.335788 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:34.315297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.0.222.62:45309)
0924 00:54:34.316074 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:34.316086 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:34.316133 (+ 47us) server_negotiation.cc:407] Connection header received
0924 00:54:34.318050 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:34.318090 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:34.318602 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:34.319100 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:34.320710 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:34.322148 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:34.323460 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:34.324142 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:34.324315 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:34.328537 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:34.328545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:34.328562 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:34.328604 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:34.331600 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:34.332341 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:34.332348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:34.332355 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:34.332548 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:34.332933 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:34.332938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:34.332941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:34.333559 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:34.333854 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:34.334329 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 00:54:34.334761 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":329,"threads_started":1}
I20260924 00:54:34.343154 19384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e268331cf19483fb460a7e6f3910645" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45309 } }
I20260924 00:54:34.343555 19384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:34.343655 19384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e268331cf19483fb460a7e6f3910645, State: Initialized, Role: FOLLOWER
I20260924 00:54:34.344202 19384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e268331cf19483fb460a7e6f3910645" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45309 } }
I20260924 00:54:34.344413 19384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:34.344542 19384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:34.344684 19384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:34.347164 19384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e268331cf19483fb460a7e6f3910645" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45309 } }
I20260924 00:54:34.347724 19384 leader_election.cc:302] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e268331cf19483fb460a7e6f3910645; no voters:
I20260924 00:54:34.348400 19384 leader_election.cc:288] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:34.348500 19389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:34.349116 19389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [term 1 LEADER]: Becoming Leader. State: Replica: 6e268331cf19483fb460a7e6f3910645, State: Running, Role: LEADER
I20260924 00:54:34.349714 19389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e268331cf19483fb460a7e6f3910645" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45309 } }
I20260924 00:54:34.350876 19384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:34.352788 19391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e268331cf19483fb460a7e6f3910645. Latest consensus state: current_term: 1 leader_uuid: "6e268331cf19483fb460a7e6f3910645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e268331cf19483fb460a7e6f3910645" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45309 } } }
I20260924 00:54:34.352835 19390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e268331cf19483fb460a7e6f3910645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e268331cf19483fb460a7e6f3910645" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45309 } } }
I20260924 00:54:34.353271 19390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:34.353343 19391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:34.358444 19395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:34.362792 19395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:34.371702 19395 catalog_manager.cc:1347] Generated new cluster ID: 2ca1f187776a404d99eda326ea43f79b
I20260924 00:54:34.371836 19395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:34.400897 19395 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:34.402402 19395 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:34.414274 19395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645: Generated new TSK 0
I20260924 00:54:34.415174 19395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:54:34.489576 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45309
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:34.827736 19410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:34.828223 19410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:34.828447 19410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:34.838758 19410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:34.838872 19410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:34.839000 19410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:54:34.849301 19410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:34.850606 19410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:34.851948 19410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:34.864988 19415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:34.865012 19418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:34.864941 19416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:34.865787 19410 server_base.cc:1034] running on GCE node
I20260924 00:54:34.866387 19410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:34.867380 19410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:34.868608 19410 hybrid_clock.cc:648] HybridClock initialized: now 1790211274868550 us; error 50 us; skew 500 ppm
I20260924 00:54:34.868916 19410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:34 dist-test-slave-rtvz krb5kdc[19303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:34.872337 19410 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:54:34.874089 19410 webserver.cc:460] Webserver started at http://127.0.222.1:40601/ using document root <none> and password file <none>
I20260924 00:54:34.874809 19410 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:34.874943 19410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:34.875268 19410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:34.878499 19410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "bd5c725e719a427c9dcd839f664e8e1a"
format_stamp: "Formatted at 2026-09-24 00:54:34 on dist-test-slave-rtvz"
server_key: "697926f5d327766f2dec4bf19d878579"
server_key_iv: "5218631633729fa8d165addf95c44eba"
server_key_version: "encryptionkey@0"
I20260924 00:54:34.879318 19410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "bd5c725e719a427c9dcd839f664e8e1a"
format_stamp: "Formatted at 2026-09-24 00:54:34 on dist-test-slave-rtvz"
server_key: "697926f5d327766f2dec4bf19d878579"
server_key_iv: "5218631633729fa8d165addf95c44eba"
server_key_version: "encryptionkey@0"
I20260924 00:54:34.884588 19410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 00:54:34.888574 19425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:34.890300 19410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:34.890493 19410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "bd5c725e719a427c9dcd839f664e8e1a"
format_stamp: "Formatted at 2026-09-24 00:54:34 on dist-test-slave-rtvz"
server_key: "697926f5d327766f2dec4bf19d878579"
server_key_iv: "5218631633729fa8d165addf95c44eba"
server_key_version: "encryptionkey@0"
I20260924 00:54:34.890727 19410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:34.929644 19410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:34.934208 19410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:34.934500 19410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:34.935511 19410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:34.936920 19410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:34.937009 19410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:34.937130 19410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:34.937182 19410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:34.970571 19410 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:37959
I20260924 00:54:34.970647 19538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:37959 every 8 connection(s)
I20260924 00:54:34.971904 19410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:54:34.973753 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 19410
I20260924 00:54:34.973937 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:54:34.974373 888 external_mini_cluster.cc:1428] Setting key 43530cdff90d5c4507c661dbb7adaf53
Sep 24 00:54:34 dist-test-slave-rtvz krb5kdc[19303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:34.994405 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:34.974768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:52245 (local address 127.0.222.62:45309)
0924 00:54:34.975290 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:34.975296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:34.977275 (+ 1979us) server_negotiation.cc:407] Connection header received
0924 00:54:34.978443 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:34.978448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:34.978545 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:34.978666 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:34.980375 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:34.981160 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:34.983404 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:34.983829 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:34.983930 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:34.987772 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:34.987778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:34.987783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:34.987826 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:34.990864 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:34.991817 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:34.991822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:34.991826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:34.991987 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:34.992446 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:34.992450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:34.992453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:34.992825 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:34.992968 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:34.993769 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 00:54:34.994022 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":260,"threads_started":1}
I20260924 00:54:34.995457 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:34.975430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45309 (local address 127.0.222.1:52245)
0924 00:54:34.976274 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:34.976325 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:34.978164 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:34.978908 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:34.978917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:34.979449 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:34.980057 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:34.980080 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:34.981426 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:34.981431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:34.983231 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:34.983249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:34.984069 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:34.984076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:34.984330 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:34.985014 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:34.985037 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:34.987502 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:34.991096 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:34.991105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:34.991122 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:34.991681 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:34.992115 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:34.992119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:34.992121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:34.992345 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:34.992948 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:34.992955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:34.993368 (+ 413us) client_negotiation.cc:769] Sending connection context
0924 00:54:34.993638 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 00:54:34.994093 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":326,"threads_started":1}
I20260924 00:54:34.997221 19539 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45309
I20260924 00:54:34.997537 19539 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:34.998286 19539 heartbeater.cc:511] Master 127.0.222.62:45309 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
I20260924 00:54:35.000552 19348 ts_manager.cc:194] Registered new tserver with Master: bd5c725e719a427c9dcd839f664e8e1a (127.0.222.1:37959)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
I20260924 00:54:35.002202 19348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:52245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:54:35.033483 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45309
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:35.370995 19545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:35.371299 19545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:35.371430 19545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:35.378819 19545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:35.378948 19545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:35.379112 19545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:54:35.389016 19545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:35.390336 19545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:35.391673 19545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:35.405100 19551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:35.405337 19550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:35.405586 19553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:35.406250 19545 server_base.cc:1034] running on GCE node
I20260924 00:54:35.406992 19545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:35.407979 19545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:35.409250 19545 hybrid_clock.cc:648] HybridClock initialized: now 1790211275409168 us; error 35 us; skew 500 ppm
I20260924 00:54:35.409569 19545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:35 dist-test-slave-rtvz krb5kdc[19303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:35.416036 19545 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:54:35.417623 19545 webserver.cc:460] Webserver started at http://127.0.222.2:40783/ using document root <none> and password file <none>
I20260924 00:54:35.418308 19545 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:35.418411 19545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:35.418680 19545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:35.421738 19545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "94ad18a1bf16488486892f8c07790ff8"
format_stamp: "Formatted at 2026-09-24 00:54:35 on dist-test-slave-rtvz"
server_key: "d70da8dc86f4b17a4b53af334b9fb8c0"
server_key_iv: "22b423411513171fa75582c25f0b5ab0"
server_key_version: "encryptionkey@0"
I20260924 00:54:35.422564 19545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "94ad18a1bf16488486892f8c07790ff8"
format_stamp: "Formatted at 2026-09-24 00:54:35 on dist-test-slave-rtvz"
server_key: "d70da8dc86f4b17a4b53af334b9fb8c0"
server_key_iv: "22b423411513171fa75582c25f0b5ab0"
server_key_version: "encryptionkey@0"
I20260924 00:54:35.427838 19545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:54:35.431504 19560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:35.433107 19545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:54:35.433305 19545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "94ad18a1bf16488486892f8c07790ff8"
format_stamp: "Formatted at 2026-09-24 00:54:35 on dist-test-slave-rtvz"
server_key: "d70da8dc86f4b17a4b53af334b9fb8c0"
server_key_iv: "22b423411513171fa75582c25f0b5ab0"
server_key_version: "encryptionkey@0"
I20260924 00:54:35.433501 19545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:35.452327 19545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:35.456723 19545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:35.457002 19545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:35.458232 19545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:35.459928 19545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:35.460016 19545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:35.460107 19545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:35.460150 19545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:35.492404 19545 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:37525
I20260924 00:54:35.492424 19673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:37525 every 8 connection(s)
I20260924 00:54:35.493788 19545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:54:35.499074 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 19545
I20260924 00:54:35.499226 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:54:35.499627 888 external_mini_cluster.cc:1428] Setting key fd2782f6acde9b506179851961b592ea
Sep 24 00:54:35 dist-test-slave-rtvz krb5kdc[19303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:35.514374 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:35.496488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:45471 (local address 127.0.222.62:45309)
0924 00:54:35.496929 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:35.496935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:35.497952 (+ 1017us) server_negotiation.cc:407] Connection header received
0924 00:54:35.499791 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:35.499796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:35.499882 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:35.500017 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:35.501873 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:35.502898 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:35.504463 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:35.505025 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:35.505137 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:35.508567 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:35.508573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:35.508578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:35.508613 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:35.511237 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:35.512116 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:35.512120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:35.512123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:35.512253 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:35.512680 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:35.512683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:35.512684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:35.512941 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:35.513056 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:35.513882 (+ 826us) server_negotiation.cc:299] Negotiation successful
0924 00:54:35.514112 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":199,"threads_started":1}
I20260924 00:54:35.515676 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:35.497062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45309 (local address 127.0.222.2:45471)
0924 00:54:35.497798 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:35.497841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:35.499526 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:35.500190 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:35.500200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:35.500917 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:35.501713 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:35.501739 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:35.503078 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:35.503082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:35.504330 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:35.504342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:35.505340 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:35.505347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:35.505560 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:35.506200 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:35.506228 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:35.508348 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:35.511383 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:35.511392 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:35.511410 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:35.511979 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:35.512347 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:35.512351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:35.512355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:35.512588 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:35.513056 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:35.513063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:35.513497 (+ 434us) client_negotiation.cc:769] Sending connection context
0924 00:54:35.513809 (+ 312us) client_negotiation.cc:240] Negotiation successful
0924 00:54:35.514340 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":292,"threads_started":1}
I20260924 00:54:35.517271 19674 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45309
I20260924 00:54:35.517601 19674 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:35.518445 19674 heartbeater.cc:511] Master 127.0.222.62:45309 requested a full tablet report, sending...
I20260924 00:54:35.520083 19348 ts_manager.cc:194] Registered new tserver with Master: 94ad18a1bf16488486892f8c07790ff8 (127.0.222.2:37525)
I20260924 00:54:35.521229 19348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:45471
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:54:35.554272 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45309
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:35.880821 19680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:35.881125 19680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:35.881268 19680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:35.888491 19680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:35.888614 19680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:35.888765 19680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:54:35.898772 19680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:35.900111 19680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:35.901520 19680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:35.914965 19685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:35.914897 19688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:35.914944 19686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:35.915795 19680 server_base.cc:1034] running on GCE node
I20260924 00:54:35.916476 19680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:35.917481 19680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:35.918708 19680 hybrid_clock.cc:648] HybridClock initialized: now 1790211275918636 us; error 60 us; skew 500 ppm
I20260924 00:54:35.919097 19680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:35 dist-test-slave-rtvz krb5kdc[19303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:35.922508 19680 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:54:35.924158 19680 webserver.cc:460] Webserver started at http://127.0.222.3:44969/ using document root <none> and password file <none>
I20260924 00:54:35.924863 19680 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:35.924981 19680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:35.925324 19680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:35.928798 19680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "11b53193947d4fc282f7828a06842fc1"
format_stamp: "Formatted at 2026-09-24 00:54:35 on dist-test-slave-rtvz"
server_key: "a790fa27a7b6d5aaabcf6919e37888ba"
server_key_iv: "255e566934310b69fb8a26bf91dd9b4c"
server_key_version: "encryptionkey@0"
I20260924 00:54:35.929950 19680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "11b53193947d4fc282f7828a06842fc1"
format_stamp: "Formatted at 2026-09-24 00:54:35 on dist-test-slave-rtvz"
server_key: "a790fa27a7b6d5aaabcf6919e37888ba"
server_key_iv: "255e566934310b69fb8a26bf91dd9b4c"
server_key_version: "encryptionkey@0"
I20260924 00:54:35.936685 19680 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 00:54:35.940268 19695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:35.941783 19680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:35.941947 19680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "11b53193947d4fc282f7828a06842fc1"
format_stamp: "Formatted at 2026-09-24 00:54:35 on dist-test-slave-rtvz"
server_key: "a790fa27a7b6d5aaabcf6919e37888ba"
server_key_iv: "255e566934310b69fb8a26bf91dd9b4c"
server_key_version: "encryptionkey@0"
I20260924 00:54:35.942266 19680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:35.955504 19680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:35.960243 19680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:35.960536 19680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:35.961571 19680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:35.963014 19680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:35.963096 19680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:35.963219 19680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:35.963272 19680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:35.996623 19680 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:39213
I20260924 00:54:35.996708 19808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:39213 every 8 connection(s)
I20260924 00:54:35.997915 19680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:54:35.998232 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 19680
I20260924 00:54:35.998406 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:54:35.998952 888 external_mini_cluster.cc:1428] Setting key 8dbad00d8d9cff8081e54333c952a290
I20260924 00:54:36.005460 19539 heartbeater.cc:503] Master 127.0.222.62:45309 was elected leader, sending a full tablet report...
Sep 24 00:54:36 dist-test-slave-rtvz krb5kdc[19303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:36.019140 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.000733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:48191 (local address 127.0.222.62:45309)
0924 00:54:36.000961 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.000968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.002318 (+ 1350us) server_negotiation.cc:407] Connection header received
0924 00:54:36.004176 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.004181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.004274 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.004355 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:36.005967 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.007025 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.008744 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.009166 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.009335 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.012914 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:36.012919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:36.012923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:36.012957 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:36.016104 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.017000 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.017004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.017007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.017139 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.017568 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.017571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.017573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.017845 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:36.017957 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.018679 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.018852 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 00:54:36.020128 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.001376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45309 (local address 127.0.222.3:48191)
0924 00:54:36.002179 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:36.002228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:36.003928 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:36.004606 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:36.004614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:36.005143 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:36.005803 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.005823 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.007197 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.007203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.008548 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.008562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.009420 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.009425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.009684 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.010346 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:36.010370 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:36.012660 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:36.016273 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:36.016284 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:36.016304 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:36.016863 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:36.017261 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:36.017264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:36.017266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:36.017460 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:36.017996 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:36.018003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:36.018338 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 00:54:36.018588 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 00:54:36.019025 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":313,"threads_started":1}
I20260924 00:54:36.021559 19809 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45309
I20260924 00:54:36.021878 19809 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:36.022632 19809 heartbeater.cc:511] Master 127.0.222.62:45309 requested a full tablet report, sending...
I20260924 00:54:36.024044 19348 ts_manager.cc:194] Registered new tserver with Master: 11b53193947d4fc282f7828a06842fc1 (127.0.222.3:39213)
I20260924 00:54:36.024858 19348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:48191
I20260924 00:54:36.027452 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:54:36.045428 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.031539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.0.222.62:45309)
0924 00:54:36.031752 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.031758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.032059 (+ 301us) server_negotiation.cc:407] Connection header received
0924 00:54:36.032222 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.032226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.032296 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.032374 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:36.033865 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.034610 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.035915 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.036356 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.036475 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.039382 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:36.039388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:36.039392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:36.039431 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:36.042334 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.043353 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.043358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.043361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.043548 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.044003 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.044007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.044011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.044333 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:36.044566 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.044915 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.045061 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:54:36.049772 19348 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:54:36.056854 19348 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:54:36.083186 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.069575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:37959 (local address 127.0.0.1:36552)
0924 00:54:36.074237 (+ 4662us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:36.074274 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:36.074467 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:36.075463 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:36.075473 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:36.075527 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:36.076151 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.076176 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.078486 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.078491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.080567 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.080579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.082308 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.082315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.082533 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.082600 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 00:54:36.082709 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:54:36.082869 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4424,"mutex_wait_us":53,"spinlock_wait_cycles":3968,"thread_start_us":3711,"threads_started":1}
I20260924 00:54:36.084547 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.070117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.0.222.1:37959)
0924 00:54:36.073858 (+ 3741us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.073867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.074509 (+ 642us) server_negotiation.cc:407] Connection header received
0924 00:54:36.074766 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.074776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.075151 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.076487 (+ 1336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:36.076763 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.078296 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.080728 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.082070 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.083711 (+ 1641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.083961 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.084046 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.084215 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3580,"thread_start_us":3387,"threads_started":1}
I20260924 00:54:36.084551 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.070371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.0.222.2:37525)
0924 00:54:36.070971 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.070982 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.075971 (+ 4989us) server_negotiation.cc:407] Connection header received
0924 00:54:36.076083 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.076091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.076417 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.077315 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:36.077571 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.079111 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.081822 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.083399 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.083603 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.083822 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.084085 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.084262 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":272,"threads_started":1}
I20260924 00:54:36.084606 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.069719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37525 (local address 127.0.0.1:48008)
0924 00:54:36.075386 (+ 5667us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:36.075405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:36.075548 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:36.076640 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:36.076644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:36.076669 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:36.077100 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.077113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.079265 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.079271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.081596 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.081613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.083683 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.083690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.083861 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.083891 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 00:54:36.083980 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:54:36.084116 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5540,"mutex_wait_us":54,"spinlock_wait_cycles":4864,"thread_start_us":3765,"threads_started":1}
I20260924 00:54:36.085335 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.074349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:39213 (local address 127.0.0.1:53008)
0924 00:54:36.076105 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:36.076121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:36.076285 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:36.077485 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:36.077489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:36.077512 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:36.077893 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.077906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.080025 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.080031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.082378 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.082394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.084525 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.084545 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.084724 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.084757 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:54:36.084824 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 00:54:36.084953 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1630,"mutex_wait_us":22,"thread_start_us":1429,"threads_started":1}
I20260924 00:54:36.086001 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.074648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.0.222.3:39213)
0924 00:54:36.076861 (+ 2213us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.076871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.076905 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:54:36.076999 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.077006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.077281 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.078091 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:36.078359 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.079867 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.082600 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.084261 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.085226 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.085457 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.085526 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.085688 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2028,"thread_start_us":1885,"threads_started":1}
I20260924 00:54:36.088814 19473 tablet_service.cc:1467] Processing CreateTablet for tablet 01508aafce4c4d2f9c4f254415b69139 (DEFAULT_TABLE table=test-table [id=703dfa00cdb24cd098c82127104ab4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:36.088814 19608 tablet_service.cc:1467] Processing CreateTablet for tablet 01508aafce4c4d2f9c4f254415b69139 (DEFAULT_TABLE table=test-table [id=703dfa00cdb24cd098c82127104ab4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:36.089709 19743 tablet_service.cc:1467] Processing CreateTablet for tablet 01508aafce4c4d2f9c4f254415b69139 (DEFAULT_TABLE table=test-table [id=703dfa00cdb24cd098c82127104ab4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:36.090409 19473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01508aafce4c4d2f9c4f254415b69139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:36.090411 19608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01508aafce4c4d2f9c4f254415b69139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:36.090893 19743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01508aafce4c4d2f9c4f254415b69139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:36.104020 19827 tablet_bootstrap.cc:492] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Bootstrap starting.
I20260924 00:54:36.105995 19828 tablet_bootstrap.cc:492] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Bootstrap starting.
I20260924 00:54:36.108587 19827 tablet_bootstrap.cc:654] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:36.110198 19827 log.cc:824] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:36.110446 19828 tablet_bootstrap.cc:654] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:36.110615 19829 tablet_bootstrap.cc:492] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Bootstrap starting.
I20260924 00:54:36.111809 19828 log.cc:824] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:36.113951 19829 tablet_bootstrap.cc:654] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:36.115275 19829 log.cc:824] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:36.115351 19828 tablet_bootstrap.cc:492] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: No bootstrap required, opened a new log
I20260924 00:54:36.115622 19828 ts_tablet_manager.cc:1397] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 00:54:36.115658 19827 tablet_bootstrap.cc:492] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: No bootstrap required, opened a new log
I20260924 00:54:36.115928 19827 ts_tablet_manager.cc:1397] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260924 00:54:36.123538 19827 raft_consensus.cc:348] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.123944 19827 raft_consensus.cc:374] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:36.124063 19827 raft_consensus.cc:729] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b53193947d4fc282f7828a06842fc1, State: Initialized, Role: FOLLOWER
I20260924 00:54:36.124715 19827 consensus_queue.cc:260] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.125105 19829 tablet_bootstrap.cc:492] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: No bootstrap required, opened a new log
I20260924 00:54:36.125525 19829 ts_tablet_manager.cc:1397] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260924 00:54:36.126169 19828 raft_consensus.cc:348] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.126588 19827 ts_tablet_manager.cc:1428] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260924 00:54:36.126663 19828 raft_consensus.cc:374] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:36.126821 19828 raft_consensus.cc:729] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5c725e719a427c9dcd839f664e8e1a, State: Initialized, Role: FOLLOWER
I20260924 00:54:36.127195 19809 heartbeater.cc:503] Master 127.0.222.62:45309 was elected leader, sending a full tablet report...
I20260924 00:54:36.129027 19828 consensus_queue.cc:260] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.130837 19828 ts_tablet_manager.cc:1428] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 00:54:36.135792 19829 raft_consensus.cc:348] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.136196 19829 raft_consensus.cc:374] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:36.136296 19829 raft_consensus.cc:729] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ad18a1bf16488486892f8c07790ff8, State: Initialized, Role: FOLLOWER
I20260924 00:54:36.136828 19829 consensus_queue.cc:260] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.137871 19674 heartbeater.cc:503] Master 127.0.222.62:45309 was elected leader, sending a full tablet report...
I20260924 00:54:36.138512 19829 ts_tablet_manager.cc:1428] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
W20260924 00:54:36.224494 19540 tablet.cc:2369] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:54:36.239504 19834 raft_consensus.cc:482] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:36.239918 19834 raft_consensus.cc:504] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.241765 19834 leader_election.cc:288] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ad18a1bf16488486892f8c07790ff8 (127.0.222.2:37525), 11b53193947d4fc282f7828a06842fc1 (127.0.222.3:39213)
W20260924 00:54:36.245798 19675 tablet.cc:2369] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:54:36.249593 19810 tablet.cc:2369] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:54:36.250000 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.242212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:53073 (local address 127.0.222.2:37525)
0924 00:54:36.242416 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.242422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.242896 (+ 474us) server_negotiation.cc:407] Connection header received
0924 00:54:36.243145 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.243156 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.243272 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.243401 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:36.244078 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.245087 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.247587 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.248841 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.249014 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.249131 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.249595 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.249739 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:54:36.250053 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.242076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:37525 (local address 127.0.222.1:53073)
0924 00:54:36.242742 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:36.242763 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:36.242950 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:36.243437 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:36.243442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:36.243467 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:36.243832 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.243846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.245250 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.245254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.247371 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.247388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.249138 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.249143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.249350 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.249392 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 00:54:36.249537 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 00:54:36.249691 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":310,"threads_started":1}
I20260924 00:54:36.250161 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.242286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:34589 (local address 127.0.222.3:39213)
0924 00:54:36.242460 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.242466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.243591 (+ 1125us) server_negotiation.cc:407] Connection header received
0924 00:54:36.243765 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.243772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.243846 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.243945 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:36.244667 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.245572 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.247964 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.249151 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.249328 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.249428 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.249782 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.249926 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:54:36.250699 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.242097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:39213 (local address 127.0.222.1:34589)
0924 00:54:36.243457 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:36.243477 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:36.243631 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:36.244076 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:36.244083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:36.244111 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:36.244479 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.244491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.245717 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.245723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.247791 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:36.247804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.249406 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.249411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:36.249602 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.249632 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 00:54:36.250352 (+ 720us) client_negotiation.cc:240] Negotiation successful
0924 00:54:36.250457 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"thread_start_us":993,"threads_started":1}
I20260924 00:54:36.251576 19628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01508aafce4c4d2f9c4f254415b69139" candidate_uuid: "bd5c725e719a427c9dcd839f664e8e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ad18a1bf16488486892f8c07790ff8" is_pre_election: true
I20260924 00:54:36.252004 19763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01508aafce4c4d2f9c4f254415b69139" candidate_uuid: "bd5c725e719a427c9dcd839f664e8e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b53193947d4fc282f7828a06842fc1" is_pre_election: true
I20260924 00:54:36.252187 19628 raft_consensus.cc:2454] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5c725e719a427c9dcd839f664e8e1a in term 0.
I20260924 00:54:36.252475 19763 raft_consensus.cc:2454] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5c725e719a427c9dcd839f664e8e1a in term 0.
I20260924 00:54:36.252902 19427 leader_election.cc:302] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ad18a1bf16488486892f8c07790ff8, bd5c725e719a427c9dcd839f664e8e1a; no voters:
I20260924 00:54:36.253307 19834 raft_consensus.cc:2781] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:54:36.253453 19834 raft_consensus.cc:482] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:54:36.253557 19834 raft_consensus.cc:3037] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:36.255789 19834 raft_consensus.cc:504] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.256426 19834 leader_election.cc:288] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [CANDIDATE]: Term 1 election: Requested vote from peers 94ad18a1bf16488486892f8c07790ff8 (127.0.222.2:37525), 11b53193947d4fc282f7828a06842fc1 (127.0.222.3:39213)
I20260924 00:54:36.256914 19628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01508aafce4c4d2f9c4f254415b69139" candidate_uuid: "bd5c725e719a427c9dcd839f664e8e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ad18a1bf16488486892f8c07790ff8"
I20260924 00:54:36.257009 19763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01508aafce4c4d2f9c4f254415b69139" candidate_uuid: "bd5c725e719a427c9dcd839f664e8e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b53193947d4fc282f7828a06842fc1"
I20260924 00:54:36.257112 19628 raft_consensus.cc:3037] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:36.257267 19763 raft_consensus.cc:3037] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:36.259369 19628 raft_consensus.cc:2454] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5c725e719a427c9dcd839f664e8e1a in term 1.
I20260924 00:54:36.259456 19763 raft_consensus.cc:2454] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5c725e719a427c9dcd839f664e8e1a in term 1.
I20260924 00:54:36.259914 19428 leader_election.cc:302] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b53193947d4fc282f7828a06842fc1, bd5c725e719a427c9dcd839f664e8e1a; no voters:
I20260924 00:54:36.260228 19834 raft_consensus.cc:2781] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:36.260648 19834 raft_consensus.cc:686] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 1 LEADER]: Becoming Leader. State: Replica: bd5c725e719a427c9dcd839f664e8e1a, State: Running, Role: LEADER
I20260924 00:54:36.261111 19834 consensus_queue.cc:237] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } }
I20260924 00:54:36.266109 19347 catalog_manager.cc:5495] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a reported cstate change: term changed from 0 to 1, leader changed from <none> to bd5c725e719a427c9dcd839f664e8e1a (127.0.222.1). New cstate: current_term: 1 leader_uuid: "bd5c725e719a427c9dcd839f664e8e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5c725e719a427c9dcd839f664e8e1a" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 } health_report { overall_health: UNKNOWN } } }
I20260924 00:54:36.327584 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.321160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.0.222.1:37959)
0924 00:54:36.321695 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.321699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.321715 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:54:36.321794 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.321797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.321870 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.321994 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:54:36.322413 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.323151 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.324906 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.325404 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.325538 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.326315 (+ 777us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:54:36.326862 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:54:36.327032 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.327281 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.327328 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260924 00:54:36.336159 19763 raft_consensus.cc:1261] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 1 FOLLOWER]: Refusing update from remote peer bd5c725e719a427c9dcd839f664e8e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:36.336159 19628 raft_consensus.cc:1261] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 1 FOLLOWER]: Refusing update from remote peer bd5c725e719a427c9dcd839f664e8e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:36.337164 19838 consensus_queue.cc:1035] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ad18a1bf16488486892f8c07790ff8" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:36.337564 19834 consensus_queue.cc:1035] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "11b53193947d4fc282f7828a06842fc1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:36.363421 19347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:36.363730 19347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:36.368943 19347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e268331cf19483fb460a7e6f3910645: Sending DeleteTablet for 3 replicas of tablet 01508aafce4c4d2f9c4f254415b69139
I20260924 00:54:36.370271 19608 tablet_service.cc:1514] Processing DeleteTablet for tablet 01508aafce4c4d2f9c4f254415b69139 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:36 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:48008
I20260924 00:54:36.370707 19473 tablet_service.cc:1514] Processing DeleteTablet for tablet 01508aafce4c4d2f9c4f254415b69139 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:36 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:36552
I20260924 00:54:36.371310 19852 tablet_replica.cc:330] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: stopping tablet replica
I20260924 00:54:36.371343 19472 tablet_service.cc:1514] Processing DeleteTablet for tablet 01508aafce4c4d2f9c4f254415b69139 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:36 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:36552
I20260924 00:54:36.371620 19853 tablet_replica.cc:330] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: stopping tablet replica
I20260924 00:54:36.371968 19852 raft_consensus.cc:2229] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:36.372006 19853 raft_consensus.cc:2229] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:54:36.372291 19608 tablet_service.cc:1514] Processing DeleteTablet for tablet 01508aafce4c4d2f9c4f254415b69139 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:36 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:48008
I20260924 00:54:36.372639 19852 raft_consensus.cc:2258] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:36.372680 19853 raft_consensus.cc:2258] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:36.373229 19743 tablet_service.cc:1514] Processing DeleteTablet for tablet 01508aafce4c4d2f9c4f254415b69139 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:36 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:53008
I20260924 00:54:36.374766 19852 ts_tablet_manager.cc:1905] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:36.377825 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 19410
I20260924 00:54:36.382341 19852 ts_tablet_manager.cc:1918] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:36.382532 19852 log.cc:1196] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/01508aafce4c4d2f9c4f254415b69139
I20260924 00:54:36.383074 19852 ts_tablet_manager.cc:1939] T 01508aafce4c4d2f9c4f254415b69139 P 94ad18a1bf16488486892f8c07790ff8: Deleting consensus metadata
I20260924 00:54:36.385160 19333 catalog_manager.cc:4841] TS 94ad18a1bf16488486892f8c07790ff8 (127.0.222.2:37525): tablet 01508aafce4c4d2f9c4f254415b69139 (table test-table [id=703dfa00cdb24cd098c82127104ab4cd]) successfully deleted
W20260924 00:54:36.385846 19333 catalog_manager.cc:4808] TS 94ad18a1bf16488486892f8c07790ff8 (127.0.222.2:37525): delete failed for tablet 01508aafce4c4d2f9c4f254415b69139 because the tablet was not found. No further retry: Not found: Tablet not found: 01508aafce4c4d2f9c4f254415b69139
I20260924 00:54:36.387917 19854 tablet_replica.cc:330] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: stopping tablet replica
I20260924 00:54:36.388397 19854 raft_consensus.cc:2229] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:36.389171 19854 raft_consensus.cc:2258] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:36.390158 19853 ts_tablet_manager.cc:1905] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:36.391510 19854 ts_tablet_manager.cc:1905] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:36.398000 19853 ts_tablet_manager.cc:1918] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:36.398232 19853 log.cc:1196] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/01508aafce4c4d2f9c4f254415b69139
I20260924 00:54:36.398415 19854 ts_tablet_manager.cc:1918] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:36.398602 19854 log.cc:1196] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/01508aafce4c4d2f9c4f254415b69139
I20260924 00:54:36.398826 19853 ts_tablet_manager.cc:1939] T 01508aafce4c4d2f9c4f254415b69139 P bd5c725e719a427c9dcd839f664e8e1a: Deleting consensus metadata
I20260924 00:54:36.399112 19854 ts_tablet_manager.cc:1939] T 01508aafce4c4d2f9c4f254415b69139 P 11b53193947d4fc282f7828a06842fc1: Deleting consensus metadata
I20260924 00:54:36.400816 19334 catalog_manager.cc:4841] TS bd5c725e719a427c9dcd839f664e8e1a (127.0.222.1:37959): tablet 01508aafce4c4d2f9c4f254415b69139 (table test-table [id=703dfa00cdb24cd098c82127104ab4cd]) successfully deleted
Sep 24 00:54:36 dist-test-slave-rtvz krb5kdc[19303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
W20260924 00:54:36.401664 19334 catalog_manager.cc:4808] TS bd5c725e719a427c9dcd839f664e8e1a (127.0.222.1:37959): delete failed for tablet 01508aafce4c4d2f9c4f254415b69139 because the tablet was not found. No further retry: Not found: Tablet not found: 01508aafce4c4d2f9c4f254415b69139
I20260924 00:54:36.401942 19334 catalog_manager.cc:4841] TS 11b53193947d4fc282f7828a06842fc1 (127.0.222.3:39213): tablet 01508aafce4c4d2f9c4f254415b69139 (table test-table [id=703dfa00cdb24cd098c82127104ab4cd]) successfully deleted
I20260924 00:54:36.408412 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.380969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.0.222.1:37959)
0924 00:54:36.381296 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.381302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.391139 (+ 9837us) server_negotiation.cc:407] Connection header received
0924 00:54:36.391605 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.391612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.391696 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.391769 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:36.393567 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.394695 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.396493 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.397217 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.397313 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.402709 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:36.402720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:36.402733 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:36.402768 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:36.405683 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.406341 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.406348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.406350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.406507 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.406870 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.406874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.406876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.407220 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:36.407555 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.407846 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.408083 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260924 00:54:36.409063 19534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:36.509032 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 19410
I20260924 00:54:36.523712 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 19545
Sep 24 00:54:36 dist-test-slave-rtvz krb5kdc[19303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:54:36.538700 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.524513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.0.222.2:37525)
0924 00:54:36.524693 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.524699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.524718 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:54:36.524843 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.524847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.524986 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.525083 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:36.526414 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.527147 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.528419 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.528808 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.528923 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.533137 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:36.533153 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:36.533169 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:36.533232 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:36.536027 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.536729 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.536739 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.536743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.536889 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.537359 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.537363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.537365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.537754 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:36.538016 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.538267 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.538438 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:54:36.539376 19669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:36.628145 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 19545
I20260924 00:54:36.642676 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 19680
Sep 24 00:54:36 dist-test-slave-rtvz krb5kdc[19303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:54:36.658771 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:36.643372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.222.3:39213)
0924 00:54:36.643547 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:36.643551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:36.643569 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:54:36.643711 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:36.643715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:36.643779 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:36.643859 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:36.645185 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.646051 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.647318 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:36.647704 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:36.647792 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:36.652190 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:36.652208 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:36.652226 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:36.652272 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:36.655288 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.656489 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.656498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.656500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.656655 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:36.657158 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:36.657162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:36.657165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:36.657786 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:36.658040 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:36.658313 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 00:54:36.658511 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:54:36.659495 19804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:36.746987 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 19680
I20260924 00:54:36.761931 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 19316
I20260924 00:54:36.763082 19379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:36.879184 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 19316
2026-09-24T00:54:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4268 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T00:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:36Z Disabled control of system clock
I20260924 00:54:36.951485 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45447
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:45447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:54:37.281968 19863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:37.282276 19863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:37.282387 19863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:37.289784 19863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:37.289911 19863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:37.289949 19863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:37.290014 19863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:37.290079 19863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:37.299573 19863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:45447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:45447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:37.300925 19863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:37.302706 19863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:37.314181 19869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:37.314206 19868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:37.314303 19872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:37.315218 19863 server_base.cc:1034] running on GCE node
I20260924 00:54:37.316027 19863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:37.317524 19863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:37.318722 19863 hybrid_clock.cc:648] HybridClock initialized: now 1790211277318661 us; error 60 us; skew 500 ppm
I20260924 00:54:37.319061 19863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:37.321341 19863 webserver.cc:460] Webserver started at http://127.0.222.62:46289/ using document root <none> and password file <none>
I20260924 00:54:37.322047 19863 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:37.322152 19863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:37.322469 19863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:37.325525 19863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c"
format_stamp: "Formatted at 2026-09-24 00:54:37 on dist-test-slave-rtvz"
server_key: "7db3c4068f5e1921694386ea0516abbc"
server_key_iv: "2a282de13bbc8991e76bfc872f0ddf87"
server_key_version: "encryptionkey@0"
I20260924 00:54:37.326340 19863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c"
format_stamp: "Formatted at 2026-09-24 00:54:37 on dist-test-slave-rtvz"
server_key: "7db3c4068f5e1921694386ea0516abbc"
server_key_iv: "2a282de13bbc8991e76bfc872f0ddf87"
server_key_version: "encryptionkey@0"
I20260924 00:54:37.332263 19863 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 00:54:37.335963 19878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:37.337438 19863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:37.337656 19863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c"
format_stamp: "Formatted at 2026-09-24 00:54:37 on dist-test-slave-rtvz"
server_key: "7db3c4068f5e1921694386ea0516abbc"
server_key_iv: "2a282de13bbc8991e76bfc872f0ddf87"
server_key_version: "encryptionkey@0"
I20260924 00:54:37.337873 19863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:37.374413 19863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:37.375294 19863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:37.375571 19863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:37.392719 19863 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:45447
I20260924 00:54:37.392737 19929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:45447 every 8 connection(s)
I20260924 00:54:37.394141 19863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:37.396409 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 19863
I20260924 00:54:37.396554 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:37.397020 888 external_mini_cluster.cc:1428] Setting key 5799ee2ca574330b4369acc02f3c8196
I20260924 00:54:37.398442 19930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:37.412667 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:37.398960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.0.222.62:45447)
0924 00:54:37.400043 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:37.400066 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:37.400118 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:54:37.402031 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:37.402062 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:37.402718 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:37.403194 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:37.403753 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:37.405802 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:37.407372 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:37.408364 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:37.408556 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:37.409686 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:37.409696 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:37.409718 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:37.410491 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:37.410609 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:37.410635 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:37.410755 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:37.410989 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 00:54:37.411362 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":333,"threads_started":1}
I20260924 00:54:37.413477 19930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c: Bootstrap starting.
I20260924 00:54:37.416635 19930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:37.417661 19930 log.cc:824] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:37.420578 19930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c: No bootstrap required, opened a new log
I20260924 00:54:37.428015 19930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45447 } }
I20260924 00:54:37.428395 19930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:37.428494 19930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5659e0a4b57a4f3a9f08d2cedd76a12c, State: Initialized, Role: FOLLOWER
I20260924 00:54:37.429095 19930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45447 } }
I20260924 00:54:37.429335 19930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:37.429467 19930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:37.429627 19930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:37.431783 19930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45447 } }
I20260924 00:54:37.432314 19930 leader_election.cc:302] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5659e0a4b57a4f3a9f08d2cedd76a12c; no voters:
I20260924 00:54:37.432940 19930 leader_election.cc:288] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:37.433111 19935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:37.433801 19935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [term 1 LEADER]: Becoming Leader. State: Replica: 5659e0a4b57a4f3a9f08d2cedd76a12c, State: Running, Role: LEADER
I20260924 00:54:37.434291 19935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45447 } }
I20260924 00:54:37.434966 19930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:37.437291 19936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45447 } } }
I20260924 00:54:37.436895 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5659e0a4b57a4f3a9f08d2cedd76a12c. Latest consensus state: current_term: 1 leader_uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5659e0a4b57a4f3a9f08d2cedd76a12c" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 45447 } } }
I20260924 00:54:37.437659 19936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:37.437731 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:37.438530 19943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:37.443611 19943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:37.456017 19943 catalog_manager.cc:1347] Generated new cluster ID: 1329bac78d5d4e10ba36ab87fd00b2cb
I20260924 00:54:37.456166 19943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:37.477614 19943 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:37.478935 19943 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:37.487048 19943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c: Generated new TSK 0
I20260924 00:54:37.487871 19943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:37.499580 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45447
--builtin_ntp_servers=127.0.222.20:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:54:37.859064 19955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:37.859370 19955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:37.859465 19955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:37.867530 19955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:37.867667 19955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:37.867837 19955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:54:37.877751 19955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:37.879103 19955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:37.880985 19955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:37.894637 19961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:37.894680 19963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:37.894707 19960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:37.895848 19955 server_base.cc:1034] running on GCE node
I20260924 00:54:37.896521 19955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:37.897643 19955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:37.898872 19955 hybrid_clock.cc:648] HybridClock initialized: now 1790211277898815 us; error 48 us; skew 500 ppm
I20260924 00:54:37.899226 19955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:37.901795 19955 webserver.cc:460] Webserver started at http://127.0.222.1:34161/ using document root <none> and password file <none>
I20260924 00:54:37.902521 19955 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:37.902624 19955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:37.902942 19955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:37.905932 19955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "5f832104ac0041a281b3cd29fc2e8bc2"
format_stamp: "Formatted at 2026-09-24 00:54:37 on dist-test-slave-rtvz"
server_key: "55aa12266db03c9f0e2bca7a47bdc181"
server_key_iv: "57cee032f78ed0a6c62c56a46ef82949"
server_key_version: "encryptionkey@0"
I20260924 00:54:37.906733 19955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "5f832104ac0041a281b3cd29fc2e8bc2"
format_stamp: "Formatted at 2026-09-24 00:54:37 on dist-test-slave-rtvz"
server_key: "55aa12266db03c9f0e2bca7a47bdc181"
server_key_iv: "57cee032f78ed0a6c62c56a46ef82949"
server_key_version: "encryptionkey@0"
I20260924 00:54:37.912393 19955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:54:37.916122 19969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:37.917618 19955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 00:54:37.917788 19955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "5f832104ac0041a281b3cd29fc2e8bc2"
format_stamp: "Formatted at 2026-09-24 00:54:37 on dist-test-slave-rtvz"
server_key: "55aa12266db03c9f0e2bca7a47bdc181"
server_key_iv: "57cee032f78ed0a6c62c56a46ef82949"
server_key_version: "encryptionkey@0"
I20260924 00:54:37.918012 19955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:37.949347 19955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:37.950273 19955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:37.950539 19955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:37.951632 19955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:37.953140 19955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:37.953262 19955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 00:54:37.953374 19955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:37.953447 19955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:37.986369 19955 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:43475
I20260924 00:54:37.986433 20081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:43475 every 8 connection(s)
I20260924 00:54:37.987793 19955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:54:37.988790 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 19955
I20260924 00:54:37.988988 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:54:37.989637 888 external_mini_cluster.cc:1428] Setting key 7f80380c479a16b52401e0506d97ebab
I20260924 00:54:37.993139 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45447
--builtin_ntp_servers=127.0.222.20:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:54:38.003485 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:37.990899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:44729 (local address 127.0.222.62:45447)
0924 00:54:37.991348 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:37.991352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:37.993257 (+ 1905us) server_negotiation.cc:407] Connection header received
0924 00:54:37.995795 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:37.995800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:37.995873 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:37.995967 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:37.997454 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:37.998201 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.000412 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.000860 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.000960 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.002591 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:38.002597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:38.002600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:38.002620 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:38.002662 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:38.002674 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:38.002751 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:38.002994 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 00:54:38.003105 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":247,"threads_started":1}
I20260924 00:54:38.004550 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:37.991683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45447 (local address 127.0.222.1:44729)
0924 00:54:37.993049 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:37.993102 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:37.995469 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:37.996282 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:37.996294 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:37.996419 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:37.997282 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:37.997307 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:37.998367 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:37.998373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.000252 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.000266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.001083 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.001090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.001442 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.002242 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:38.002245 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:38.002296 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:38.002387 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:38.002762 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:38.002773 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:38.002782 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:54:38.002947 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 00:54:38.003382 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":372,"threads_started":1}
I20260924 00:54:38.006259 20082 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45447
I20260924 00:54:38.006567 20082 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:38.007360 20082 heartbeater.cc:511] Master 127.0.222.62:45447 requested a full tablet report, sending...
I20260924 00:54:38.009688 19895 ts_manager.cc:194] Registered new tserver with Master: 5f832104ac0041a281b3cd29fc2e8bc2 (127.0.222.1:43475)
I20260924 00:54:38.011226 19895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.1:44729
W20260924 00:54:38.330842 20086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:38.331269 20086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:38.331386 20086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:38.339562 20086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:38.339670 20086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:38.339787 20086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:54:38.349243 20086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:38.350535 20086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:38.352264 20086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:38.365523 20091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:38.365584 20094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:38.365562 20092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:38.366384 20086 server_base.cc:1034] running on GCE node
I20260924 00:54:38.367050 20086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:38.368072 20086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:38.369305 20086 hybrid_clock.cc:648] HybridClock initialized: now 1790211278369250 us; error 48 us; skew 500 ppm
I20260924 00:54:38.369648 20086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:38.372135 20086 webserver.cc:460] Webserver started at http://127.0.222.2:33173/ using document root <none> and password file <none>
I20260924 00:54:38.372892 20086 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:38.373010 20086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:38.373381 20086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:38.376386 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3"
format_stamp: "Formatted at 2026-09-24 00:54:38 on dist-test-slave-rtvz"
server_key: "a643730401c9ed4c107885ee8ed40daf"
server_key_iv: "af224c3b2dbf36f6403212e605016607"
server_key_version: "encryptionkey@0"
I20260924 00:54:38.377224 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3"
format_stamp: "Formatted at 2026-09-24 00:54:38 on dist-test-slave-rtvz"
server_key: "a643730401c9ed4c107885ee8ed40daf"
server_key_iv: "af224c3b2dbf36f6403212e605016607"
server_key_version: "encryptionkey@0"
I20260924 00:54:38.382843 20086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 00:54:38.386616 20101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:38.388127 20086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:38.388311 20086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3"
format_stamp: "Formatted at 2026-09-24 00:54:38 on dist-test-slave-rtvz"
server_key: "a643730401c9ed4c107885ee8ed40daf"
server_key_iv: "af224c3b2dbf36f6403212e605016607"
server_key_version: "encryptionkey@0"
I20260924 00:54:38.388526 20086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:38.411134 20086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:38.412056 20086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:38.412361 20086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:38.413516 20086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:38.415234 20086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:38.415344 20086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:38.415469 20086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:38.415546 20086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:38.449263 20086 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:39663
I20260924 00:54:38.449407 20216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:39663 every 8 connection(s)
I20260924 00:54:38.450610 20086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:54:38.450896 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 20086
I20260924 00:54:38.451073 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:54:38.451607 888 external_mini_cluster.cc:1428] Setting key 8c69592e2be3c7663a52afc4a4fe2785
I20260924 00:54:38.454712 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45447
--builtin_ntp_servers=127.0.222.20:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:54:38.464988 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.453272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:59177 (local address 127.0.222.62:45447)
0924 00:54:38.453470 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:38.453476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:38.455269 (+ 1793us) server_negotiation.cc:407] Connection header received
0924 00:54:38.457737 (+ 2468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:38.457743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:38.457817 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:38.457897 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:38.459263 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.460014 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.462167 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.462604 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.462706 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.464178 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:38.464184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:38.464187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:38.464207 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:38.464242 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:38.464254 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:38.464327 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:38.464564 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 00:54:38.464671 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 00:54:38.466114 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.453929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45447 (local address 127.0.222.2:59177)
0924 00:54:38.455093 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:38.455138 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:38.457409 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:38.458159 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:38.458170 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:38.458289 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:38.459077 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.459101 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.460153 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.460158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.462010 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.462024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.462820 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.462827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.463128 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.463829 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:38.463832 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:38.463888 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:38.463979 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:38.464340 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:38.464351 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:38.464360 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:54:38.464536 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 00:54:38.464986 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":318,"threads_started":1}
I20260924 00:54:38.467658 20217 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45447
I20260924 00:54:38.468034 20217 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:38.468814 20217 heartbeater.cc:511] Master 127.0.222.62:45447 requested a full tablet report, sending...
I20260924 00:54:38.470373 19895 ts_manager.cc:194] Registered new tserver with Master: fb66cf037c4e4ca0b34f8d3f05cd00d3 (127.0.222.2:39663)
I20260924 00:54:38.471410 19895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.2:59177
W20260924 00:54:38.787242 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:38.787559 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:38.787645 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:38.795208 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:38.795323 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:38.795432 20220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:54:38.804800 20220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:45447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:38.806123 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:38.807888 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:38.821069 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:38.821220 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:38.821130 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:38.825239 20220 server_base.cc:1034] running on GCE node
I20260924 00:54:38.826090 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:38.827881 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:38.829155 20220 hybrid_clock.cc:648] HybridClock initialized: now 1790211278829089 us; error 55 us; skew 500 ppm
I20260924 00:54:38.829592 20220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:38.832197 20220 webserver.cc:460] Webserver started at http://127.0.222.3:39539/ using document root <none> and password file <none>
I20260924 00:54:38.833014 20220 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:38.833137 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:38.833496 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:38.836750 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "4f30d65f123946c5b95f1210893c86f1"
format_stamp: "Formatted at 2026-09-24 00:54:38 on dist-test-slave-rtvz"
server_key: "02e5826f7b23913fd1740b997d1856f2"
server_key_iv: "a96258d4ff0155201a7548ceda112e5b"
server_key_version: "encryptionkey@0"
I20260924 00:54:38.837574 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "4f30d65f123946c5b95f1210893c86f1"
format_stamp: "Formatted at 2026-09-24 00:54:38 on dist-test-slave-rtvz"
server_key: "02e5826f7b23913fd1740b997d1856f2"
server_key_iv: "a96258d4ff0155201a7548ceda112e5b"
server_key_version: "encryptionkey@0"
I20260924 00:54:38.843756 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 00:54:38.847548 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:38.848963 20220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:54:38.849133 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "4f30d65f123946c5b95f1210893c86f1"
format_stamp: "Formatted at 2026-09-24 00:54:38 on dist-test-slave-rtvz"
server_key: "02e5826f7b23913fd1740b997d1856f2"
server_key_iv: "a96258d4ff0155201a7548ceda112e5b"
server_key_version: "encryptionkey@0"
I20260924 00:54:38.849412 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:38.865746 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:38.866641 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:38.866926 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:38.867991 20220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:38.869532 20220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:38.869634 20220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:38.869737 20220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:38.869804 20220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:38.901150 20220 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:44483
I20260924 00:54:38.901247 20347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:44483 every 8 connection(s)
I20260924 00:54:38.902536 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:54:38.912585 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 20220
I20260924 00:54:38.912753 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:54:38.913309 888 external_mini_cluster.cc:1428] Setting key 28cfa8455109bb15fb5e21b357327cd8
I20260924 00:54:38.915488 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.905260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:58217 (local address 127.0.222.62:45447)
0924 00:54:38.905437 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:38.905441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:38.906847 (+ 1406us) server_negotiation.cc:407] Connection header received
0924 00:54:38.908680 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:38.908687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:38.908775 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:38.908878 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:38.910074 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.910867 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.912559 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.913028 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.913142 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.914670 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:38.914675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:38.914678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:38.914696 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:38.914731 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:38.914742 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:38.914815 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:38.915050 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 00:54:38.915154 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:54:38.916651 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.905722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:45447 (local address 127.0.222.3:58217)
0924 00:54:38.906679 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:38.906716 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:38.908381 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:38.909116 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:38.909125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:38.909264 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:38.909883 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.909908 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.911037 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.911041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.912403 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.912418 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.913286 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.913295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.913635 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.914329 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:38.914332 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:38.914383 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:38.914475 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:38.914829 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:38.914840 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:38.914849 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:54:38.915003 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 00:54:38.915446 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":289,"threads_started":1}
I20260924 00:54:38.918031 20348 heartbeater.cc:347] Connected to a master server at 127.0.222.62:45447
I20260924 00:54:38.918318 20348 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:38.919049 20348 heartbeater.cc:511] Master 127.0.222.62:45447 requested a full tablet report, sending...
I20260924 00:54:38.920403 19895 ts_manager.cc:194] Registered new tserver with Master: 4f30d65f123946c5b95f1210893c86f1 (127.0.222.3:44483)
I20260924 00:54:38.921226 19895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.3:58217
I20260924 00:54:38.928905 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:54:38.939417 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.933320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.0.222.62:45447)
0924 00:54:38.933529 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:38.933534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:38.934021 (+ 487us) server_negotiation.cc:407] Connection header received
0924 00:54:38.934148 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:38.934152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:38.934231 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:38.934309 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:38.934847 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.935640 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.937061 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.937520 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.937613 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.938585 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:38.938590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:38.938593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:38.938611 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:38.938645 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:38.938656 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:38.938784 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:38.939117 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 00:54:38.939167 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:54:38.943580 19895 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:54:38.950693 19895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:54:38.975579 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.965488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.0.222.2:39663)
0924 00:54:38.966969 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:38.966979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:38.967759 (+ 780us) server_negotiation.cc:407] Connection header received
0924 00:54:38.967854 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:38.967860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:38.968046 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:38.968209 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:38.969123 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.970599 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.972899 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.974403 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.974535 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.974808 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:38.975043 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 00:54:38.975219 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":1179,"threads_started":1}
I20260924 00:54:38.975612 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.964843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:39663 (local address 127.0.0.1:50454)
0924 00:54:38.967647 (+ 2804us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:38.967660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:38.967757 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:38.968430 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:38.968435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:38.968463 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:38.968910 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.968924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.970751 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.970756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.972679 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.972695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.974597 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.974604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.974811 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.974866 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 00:54:38.974982 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 00:54:38.975204 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2700,"mutex_wait_us":130,"thread_start_us":2570,"threads_started":1}
I20260924 00:54:38.976331 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.964267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:44483 (local address 127.0.0.1:51194)
0924 00:54:38.965976 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:38.965994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:38.966109 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:38.968196 (+ 2087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:38.968204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:38.968228 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:38.968696 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.968706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.970445 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.970450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.973397 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.973415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.975622 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.975628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.975780 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.975809 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 00:54:38.975880 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:54:38.975999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1561,"mutex_wait_us":231,"thread_start_us":745,"threads_started":1}
I20260924 00:54:38.976800 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.963940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43475 (local address 127.0.0.1:39830)
0924 00:54:38.965116 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:38.965162 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:38.965398 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:38.967951 (+ 2553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:38.967958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:38.967996 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:38.968417 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.968431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.970520 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.970524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.973044 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:38.973060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.975295 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.975302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:38.975504 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.975536 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 00:54:38.975631 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 00:54:38.976493 (+ 862us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":125,"thread_start_us":1761,"threads_started":1}
I20260924 00:54:38.977290 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.964676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.0.222.3:44483)
0924 00:54:38.967621 (+ 2945us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:38.967627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:38.967655 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:54:38.967749 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:38.967756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:38.968012 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:38.968209 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:38.968857 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.970307 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.973653 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.975107 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.975237 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.975487 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:38.976700 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0924 00:54:38.976908 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2802,"thread_start_us":2651,"threads_started":1}
I20260924 00:54:38.977682 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:38.964170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.0.222.1:43475)
0924 00:54:38.966623 (+ 2453us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:38.966633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:38.966680 (+ 47us) server_negotiation.cc:407] Connection header received
0924 00:54:38.966806 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:38.966815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:38.967102 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:38.967929 (+ 827us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:38.968643 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.970343 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.973310 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:38.974774 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:38.974910 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:38.975162 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:38.977164 (+ 2002us) server_negotiation.cc:299] Negotiation successful
0924 00:54:38.977372 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2245,"thread_start_us":2133,"threads_started":1}
I20260924 00:54:38.980273 20151 tablet_service.cc:1467] Processing CreateTablet for tablet 469f3d1dfeb54cbcabcc51f014f2941c (DEFAULT_TABLE table=test-table [id=18d076958a1a43cbab121f5e40e63606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:38.980947 20283 tablet_service.cc:1467] Processing CreateTablet for tablet 469f3d1dfeb54cbcabcc51f014f2941c (DEFAULT_TABLE table=test-table [id=18d076958a1a43cbab121f5e40e63606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:38.981702 20151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469f3d1dfeb54cbcabcc51f014f2941c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:38.981729 20017 tablet_service.cc:1467] Processing CreateTablet for tablet 469f3d1dfeb54cbcabcc51f014f2941c (DEFAULT_TABLE table=test-table [id=18d076958a1a43cbab121f5e40e63606]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:38.982492 20283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469f3d1dfeb54cbcabcc51f014f2941c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:38.983031 20017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469f3d1dfeb54cbcabcc51f014f2941c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:38.997711 20366 tablet_bootstrap.cc:492] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Bootstrap starting.
I20260924 00:54:39.000243 20367 tablet_bootstrap.cc:492] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: Bootstrap starting.
I20260924 00:54:39.001823 20366 tablet_bootstrap.cc:654] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:39.003324 20367 tablet_bootstrap.cc:654] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:39.003298 20366 log.cc:824] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:39.004468 20367 log.cc:824] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:39.004786 20368 tablet_bootstrap.cc:492] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: Bootstrap starting.
I20260924 00:54:39.006837 20366 tablet_bootstrap.cc:492] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: No bootstrap required, opened a new log
I20260924 00:54:39.007063 20367 tablet_bootstrap.cc:492] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: No bootstrap required, opened a new log
I20260924 00:54:39.007220 20366 ts_tablet_manager.cc:1397] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 00:54:39.007309 20367 ts_tablet_manager.cc:1397] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 00:54:39.009156 20368 tablet_bootstrap.cc:654] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:39.010432 20368 log.cc:824] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:39.013909 20368 tablet_bootstrap.cc:492] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: No bootstrap required, opened a new log
I20260924 00:54:39.014211 20368 ts_tablet_manager.cc:1397] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 00:54:39.014533 20082 heartbeater.cc:503] Master 127.0.222.62:45447 was elected leader, sending a full tablet report...
I20260924 00:54:39.015909 20367 raft_consensus.cc:348] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.016381 20367 raft_consensus.cc:374] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:39.016489 20367 raft_consensus.cc:729] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f832104ac0041a281b3cd29fc2e8bc2, State: Initialized, Role: FOLLOWER
I20260924 00:54:39.017268 20367 consensus_queue.cc:260] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.018829 20366 raft_consensus.cc:348] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.019390 20366 raft_consensus.cc:374] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:39.019537 20366 raft_consensus.cc:729] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f30d65f123946c5b95f1210893c86f1, State: Initialized, Role: FOLLOWER
I20260924 00:54:39.021008 20367 ts_tablet_manager.cc:1428] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 00:54:39.020884 20366 consensus_queue.cc:260] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.023880 20348 heartbeater.cc:503] Master 127.0.222.62:45447 was elected leader, sending a full tablet report...
I20260924 00:54:39.024034 20366 ts_tablet_manager.cc:1428] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260924 00:54:39.024480 20368 raft_consensus.cc:348] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.024938 20368 raft_consensus.cc:374] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:39.025050 20368 raft_consensus.cc:729] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb66cf037c4e4ca0b34f8d3f05cd00d3, State: Initialized, Role: FOLLOWER
I20260924 00:54:39.025691 20368 consensus_queue.cc:260] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.026749 20217 heartbeater.cc:503] Master 127.0.222.62:45447 was elected leader, sending a full tablet report...
I20260924 00:54:39.027274 20368 ts_tablet_manager.cc:1428] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 00:54:39.115828 20373 raft_consensus.cc:482] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:39.116123 20373 raft_consensus.cc:504] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.118083 20373 leader_election.cc:288] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f832104ac0041a281b3cd29fc2e8bc2 (127.0.222.1:43475), fb66cf037c4e4ca0b34f8d3f05cd00d3 (127.0.222.2:39663)
I20260924 00:54:39.123677 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:39.118397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:39663 (local address 127.0.222.3:50853)
0924 00:54:39.118669 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:39.118689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:39.118823 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:39.119244 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:39.119249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:39.119277 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:39.119457 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:39.119465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.120440 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.120443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:39.121858 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:39.121870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.122957 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.122963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:39.123118 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:39.123142 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 00:54:39.123218 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:54:39.123355 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"spinlock_wait_cycles":29952}
I20260924 00:54:39.123667 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:39.118694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:50853 (local address 127.0.222.2:39663)
0924 00:54:39.118879 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:39.118885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:39.118903 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:54:39.118978 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:39.118983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:39.119070 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:39.119148 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:39.119603 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.120310 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.122007 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.122811 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.122924 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:39.123013 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:39.123255 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 00:54:39.123378 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:54:39.124320 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:39.118449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:50801 (local address 127.0.222.1:43475)
0924 00:54:39.118675 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:39.118682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:39.119295 (+ 613us) server_negotiation.cc:407] Connection header received
0924 00:54:39.119466 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:39.119472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:39.119545 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:39.119617 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:39.120153 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.120828 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.122578 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.123473 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.123589 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:39.123682 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:39.123998 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 00:54:39.124131 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:54:39.124806 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:39.118458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:43475 (local address 127.0.222.3:50801)
0924 00:54:39.119177 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:39.119192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:39.119323 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:39.119649 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:39.119653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:39.119675 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:39.119991 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:39.119999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.120984 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.120988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:39.122410 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:39.122421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.123616 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.123621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:39.123809 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:39.123844 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:54:39.124487 (+ 643us) client_negotiation.cc:240] Negotiation successful
0924 00:54:39.124602 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":382,"threads_started":1}
I20260924 00:54:39.124696 20171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469f3d1dfeb54cbcabcc51f014f2941c" candidate_uuid: "4f30d65f123946c5b95f1210893c86f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" is_pre_election: true
I20260924 00:54:39.125267 20171 raft_consensus.cc:2454] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f30d65f123946c5b95f1210893c86f1 in term 0.
I20260924 00:54:39.125995 20236 leader_election.cc:302] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f30d65f123946c5b95f1210893c86f1, fb66cf037c4e4ca0b34f8d3f05cd00d3; no voters:
I20260924 00:54:39.125854 20037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469f3d1dfeb54cbcabcc51f014f2941c" candidate_uuid: "4f30d65f123946c5b95f1210893c86f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" is_pre_election: true
I20260924 00:54:39.126351 20373 raft_consensus.cc:2781] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:54:39.126396 20037 raft_consensus.cc:2454] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f30d65f123946c5b95f1210893c86f1 in term 0.
I20260924 00:54:39.126513 20373 raft_consensus.cc:482] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:54:39.126605 20373 raft_consensus.cc:3037] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:39.128846 20373 raft_consensus.cc:504] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.129453 20373 leader_election.cc:288] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [CANDIDATE]: Term 1 election: Requested vote from peers 5f832104ac0041a281b3cd29fc2e8bc2 (127.0.222.1:43475), fb66cf037c4e4ca0b34f8d3f05cd00d3 (127.0.222.2:39663)
I20260924 00:54:39.129805 20037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469f3d1dfeb54cbcabcc51f014f2941c" candidate_uuid: "4f30d65f123946c5b95f1210893c86f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f832104ac0041a281b3cd29fc2e8bc2"
I20260924 00:54:39.130012 20037 raft_consensus.cc:3037] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:39.130002 20171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469f3d1dfeb54cbcabcc51f014f2941c" candidate_uuid: "4f30d65f123946c5b95f1210893c86f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3"
I20260924 00:54:39.130203 20171 raft_consensus.cc:3037] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:39.133265 20037 raft_consensus.cc:2454] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f30d65f123946c5b95f1210893c86f1 in term 1.
I20260924 00:54:39.133279 20171 raft_consensus.cc:2454] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f30d65f123946c5b95f1210893c86f1 in term 1.
I20260924 00:54:39.133701 20236 leader_election.cc:302] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f30d65f123946c5b95f1210893c86f1, fb66cf037c4e4ca0b34f8d3f05cd00d3; no voters:
I20260924 00:54:39.134012 20373 raft_consensus.cc:2781] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:39.134438 20373 raft_consensus.cc:686] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 1 LEADER]: Becoming Leader. State: Replica: 4f30d65f123946c5b95f1210893c86f1, State: Running, Role: LEADER
I20260924 00:54:39.134913 20373 consensus_queue.cc:237] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } }
I20260924 00:54:39.139729 19895 catalog_manager.cc:5495] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f30d65f123946c5b95f1210893c86f1 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "4f30d65f123946c5b95f1210893c86f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f30d65f123946c5b95f1210893c86f1" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 44483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 } health_report { overall_health: UNKNOWN } } }
W20260924 00:54:39.154199 20349 tablet.cc:2369] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:54:39.175940 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:39.170528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.0.222.3:44483)
0924 00:54:39.170681 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:39.170686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:39.170821 (+ 135us) server_negotiation.cc:407] Connection header received
0924 00:54:39.170998 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:39.171003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:39.171146 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:39.171304 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:54:39.171684 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.172433 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.173862 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.174285 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.174381 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:39.174823 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:54:39.175387 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:54:39.175534 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:39.175712 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 00:54:39.175749 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:54:39.184580 20171 raft_consensus.cc:1261] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 1 FOLLOWER]: Refusing update from remote peer 4f30d65f123946c5b95f1210893c86f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:39.184580 20037 raft_consensus.cc:1261] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 1 FOLLOWER]: Refusing update from remote peer 4f30d65f123946c5b95f1210893c86f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:39.185688 20376 consensus_queue.cc:1035] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb66cf037c4e4ca0b34f8d3f05cd00d3" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:39.186084 20373 consensus_queue.cc:1035] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f832104ac0041a281b3cd29fc2e8bc2" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:39.215529 19894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:39.215911 19894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:39.220158 19894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5659e0a4b57a4f3a9f08d2cedd76a12c: Sending DeleteTablet for 3 replicas of tablet 469f3d1dfeb54cbcabcc51f014f2941c
I20260924 00:54:39.221452 20283 tablet_service.cc:1514] Processing DeleteTablet for tablet 469f3d1dfeb54cbcabcc51f014f2941c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:39 UTC) from {username='slave'} at 127.0.0.1:51194
I20260924 00:54:39.221457 20151 tablet_service.cc:1514] Processing DeleteTablet for tablet 469f3d1dfeb54cbcabcc51f014f2941c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:39 UTC) from {username='slave'} at 127.0.0.1:50454
I20260924 00:54:39.221496 20017 tablet_service.cc:1514] Processing DeleteTablet for tablet 469f3d1dfeb54cbcabcc51f014f2941c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:39 UTC) from {username='slave'} at 127.0.0.1:39830
I20260924 00:54:39.222491 20392 tablet_replica.cc:330] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: stopping tablet replica
I20260924 00:54:39.223078 20392 raft_consensus.cc:2229] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:39.223727 20392 raft_consensus.cc:2258] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:39.223755 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 19955
I20260924 00:54:39.226297 20392 ts_tablet_manager.cc:1905] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:39.233759 20390 tablet_replica.cc:330] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: stopping tablet replica
I20260924 00:54:39.234324 20390 raft_consensus.cc:2229] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:39.234925 20390 raft_consensus.cc:2258] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:39.235080 20392 ts_tablet_manager.cc:1918] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:39.235386 20392 log.cc:1196] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/469f3d1dfeb54cbcabcc51f014f2941c
I20260924 00:54:39.236236 20392 ts_tablet_manager.cc:1939] T 469f3d1dfeb54cbcabcc51f014f2941c P 5f832104ac0041a281b3cd29fc2e8bc2: Deleting consensus metadata
I20260924 00:54:39.238549 19879 catalog_manager.cc:4841] TS 5f832104ac0041a281b3cd29fc2e8bc2 (127.0.222.1:43475): tablet 469f3d1dfeb54cbcabcc51f014f2941c (table test-table [id=18d076958a1a43cbab121f5e40e63606]) successfully deleted
I20260924 00:54:39.239429 20390 ts_tablet_manager.cc:1905] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:39.242378 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:39.225318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.0.222.1:43475)
0924 00:54:39.225516 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:39.225523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:39.234005 (+ 8482us) server_negotiation.cc:407] Connection header received
0924 00:54:39.234291 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:39.234297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:39.234428 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:39.234510 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:39.235211 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.236497 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.238475 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.239051 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.239147 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:39.240628 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:39.240638 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:39.240657 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:39.241506 (+ 849us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:39.241570 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:39.241597 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:39.241727 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:39.241885 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:54:39.242005 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 00:54:39.242558 20391 tablet_replica.cc:330] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: stopping tablet replica
I20260924 00:54:39.243010 20391 raft_consensus.cc:2229] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:54:39.243108 20077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:39.243789 20391 raft_consensus.cc:2258] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:39.246512 20391 ts_tablet_manager.cc:1905] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:39.246630 20390 ts_tablet_manager.cc:1918] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:39.246794 20390 log.cc:1196] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/469f3d1dfeb54cbcabcc51f014f2941c
I20260924 00:54:39.247325 20390 ts_tablet_manager.cc:1939] T 469f3d1dfeb54cbcabcc51f014f2941c P fb66cf037c4e4ca0b34f8d3f05cd00d3: Deleting consensus metadata
I20260924 00:54:39.249176 19880 catalog_manager.cc:4841] TS fb66cf037c4e4ca0b34f8d3f05cd00d3 (127.0.222.2:39663): tablet 469f3d1dfeb54cbcabcc51f014f2941c (table test-table [id=18d076958a1a43cbab121f5e40e63606]) successfully deleted
I20260924 00:54:39.254542 20391 ts_tablet_manager.cc:1918] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:39.254715 20391 log.cc:1196] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/469f3d1dfeb54cbcabcc51f014f2941c
I20260924 00:54:39.255227 20391 ts_tablet_manager.cc:1939] T 469f3d1dfeb54cbcabcc51f014f2941c P 4f30d65f123946c5b95f1210893c86f1: Deleting consensus metadata
I20260924 00:54:39.257007 19882 catalog_manager.cc:4841] TS 4f30d65f123946c5b95f1210893c86f1 (127.0.222.3:44483): tablet 469f3d1dfeb54cbcabcc51f014f2941c (table test-table [id=18d076958a1a43cbab121f5e40e63606]) successfully deleted
I20260924 00:54:39.341801 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 19955
I20260924 00:54:39.363058 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 20086
I20260924 00:54:39.372143 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:39.363919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.0.222.2:39663)
0924 00:54:39.364334 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:39.364341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:39.364479 (+ 138us) server_negotiation.cc:407] Connection header received
0924 00:54:39.364562 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:39.364568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:39.364720 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:39.364843 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:39.365659 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.366845 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.368232 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.368910 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.369051 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:39.370328 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:39.370342 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:39.370364 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:39.371172 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:39.371241 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:39.371274 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:39.371425 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:39.371707 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 00:54:39.371850 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260924 00:54:39.372864 20212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:39.476682 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 20086
I20260924 00:54:39.491835 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 20220
I20260924 00:54:39.500268 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:39.492709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.0.222.3:44483)
0924 00:54:39.493247 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:39.493253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:39.493274 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:54:39.493339 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:39.493344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:39.493455 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:39.493567 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:39.493978 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.495212 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.496583 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:39.497291 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:39.497423 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:39.498527 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:39.498539 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:39.498561 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:39.499438 (+ 877us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:39.499501 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:39.499528 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:39.499643 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:39.499775 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 00:54:39.499876 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372}
I20260924 00:54:39.501034 20343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:39.622244 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 20220
I20260924 00:54:39.638751 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 19863
I20260924 00:54:39.639860 19925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:39.747169 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 19863
2026-09-24T00:54:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2866 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:39 dist-test-slave-rtvz krb5kdc[20400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:39 dist-test-slave-rtvz krb5kdc[20400](info): set up 2 sockets
Sep 24 00:54:39 dist-test-slave-rtvz krb5kdc[20400](info): commencing operation
krb5kdc: starting...
W20260924 00:54:40.776964 888 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:40 dist-test-slave-rtvz krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:54:41.040812 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46361
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:41.385038 20413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:41.385421 20413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:41.385532 20413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:41.393038 20413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:41.393152 20413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:41.393224 20413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:41.393294 20413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:41.393338 20413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:41.403515 20413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:41.404882 20413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:41.406219 20413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:41.418560 20420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:41.418604 20422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:41.418627 20419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:41.419183 20413 server_base.cc:1034] running on GCE node
I20260924 00:54:41.420178 20413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:41.421748 20413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:41.423105 20413 hybrid_clock.cc:648] HybridClock initialized: now 1790211281423035 us; error 52 us; skew 500 ppm
I20260924 00:54:41.423477 20413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:41 dist-test-slave-rtvz krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:41.426805 20413 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:54:41.428424 20413 webserver.cc:460] Webserver started at http://127.0.222.62:37545/ using document root <none> and password file <none>
I20260924 00:54:41.429136 20413 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:41.429283 20413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:41.429608 20413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:41.432623 20413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "38ef7b085748468bbef138bb2cfb7612"
format_stamp: "Formatted at 2026-09-24 00:54:41 on dist-test-slave-rtvz"
server_key: "7173459e08db3b7f620f54655fb6d7e4"
server_key_iv: "cd9428dde443612969819e9d43138036"
server_key_version: "encryptionkey@0"
I20260924 00:54:41.433408 20413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "38ef7b085748468bbef138bb2cfb7612"
format_stamp: "Formatted at 2026-09-24 00:54:41 on dist-test-slave-rtvz"
server_key: "7173459e08db3b7f620f54655fb6d7e4"
server_key_iv: "cd9428dde443612969819e9d43138036"
server_key_version: "encryptionkey@0"
I20260924 00:54:41.438712 20413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 00:54:41.442353 20429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:41.443747 20413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:54:41.443917 20413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "38ef7b085748468bbef138bb2cfb7612"
format_stamp: "Formatted at 2026-09-24 00:54:41 on dist-test-slave-rtvz"
server_key: "7173459e08db3b7f620f54655fb6d7e4"
server_key_iv: "cd9428dde443612969819e9d43138036"
server_key_version: "encryptionkey@0"
I20260924 00:54:41.444116 20413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:41.474179 20413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:41.478638 20413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:41.478888 20413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:41.496327 20413 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:46361
I20260924 00:54:41.496366 20480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:46361 every 8 connection(s)
I20260924 00:54:41.497717 20413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:41.501515 20481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:41.505733 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 20413
I20260924 00:54:41.505884 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:41.506325 888 external_mini_cluster.cc:1428] Setting key 5b596fb422f1115548257e4f759cfdce
I20260924 00:54:41.514163 20481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612: Bootstrap starting.
I20260924 00:54:41.517966 20481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:41.519270 20481 log.cc:824] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:41.522521 20481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612: No bootstrap required, opened a new log
Sep 24 00:54:41 dist-test-slave-rtvz krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:41.533053 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:41.508354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.0.222.62:46361)
0924 00:54:41.509125 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:41.509142 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:41.509220 (+ 78us) server_negotiation.cc:407] Connection header received
0924 00:54:41.512081 (+ 2861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:41.512120 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:41.512723 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:41.513240 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:41.514842 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:41.516858 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:41.518368 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:41.519455 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:41.519665 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:41.523849 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:41.523862 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:41.523883 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:41.523933 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:41.528168 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:41.528979 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:41.528990 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:41.529000 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:41.529242 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:41.529694 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:41.529699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:41.529703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:41.530452 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:41.530760 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:41.531177 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 00:54:41.531662 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":316,"threads_started":1}
I20260924 00:54:41.532732 20481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ef7b085748468bbef138bb2cfb7612" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46361 } }
I20260924 00:54:41.533234 20481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:41.533346 20481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ef7b085748468bbef138bb2cfb7612, State: Initialized, Role: FOLLOWER
I20260924 00:54:41.534065 20481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ef7b085748468bbef138bb2cfb7612" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46361 } }
I20260924 00:54:41.534353 20481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:41.534500 20481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:41.534679 20481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:41.537071 20481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ef7b085748468bbef138bb2cfb7612" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46361 } }
I20260924 00:54:41.537608 20481 leader_election.cc:302] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38ef7b085748468bbef138bb2cfb7612; no voters:
I20260924 00:54:41.538174 20481 leader_election.cc:288] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:41.538288 20486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:41.538889 20486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [term 1 LEADER]: Becoming Leader. State: Replica: 38ef7b085748468bbef138bb2cfb7612, State: Running, Role: LEADER
I20260924 00:54:41.539410 20486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ef7b085748468bbef138bb2cfb7612" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46361 } }
I20260924 00:54:41.540125 20481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:41.542210 20487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38ef7b085748468bbef138bb2cfb7612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ef7b085748468bbef138bb2cfb7612" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46361 } } }
I20260924 00:54:41.542519 20487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:41.543156 20488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ef7b085748468bbef138bb2cfb7612. Latest consensus state: current_term: 1 leader_uuid: "38ef7b085748468bbef138bb2cfb7612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38ef7b085748468bbef138bb2cfb7612" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46361 } } }
I20260924 00:54:41.543411 20488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:41.543970 20491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:41.550037 20491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:41.559854 20491 catalog_manager.cc:1347] Generated new cluster ID: d6eb969317b743ea9072e944bd3875b5
I20260924 00:54:41.559975 20491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:41.574307 20491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:41.575688 20491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:41.584808 20491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612: Generated new TSK 0
I20260924 00:54:41.585764 20491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.222.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
Entry for principal kudu/127.0.222.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab.
I20260924 00:54:41.659953 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46361
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:42.001010 20507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:42.001407 20507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:42.001530 20507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:42.009018 20507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:42.009145 20507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:42.009326 20507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:54:42.019650 20507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.1.keytab
--principal=kudu/127.0.222.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:42.020920 20507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:42.022315 20507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:42.035531 20513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:42.035555 20512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:42.035486 20515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:42.036231 20507 server_base.cc:1034] running on GCE node
I20260924 00:54:42.036912 20507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:42.037904 20507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:42.039140 20507 hybrid_clock.cc:648] HybridClock initialized: now 1790211282039085 us; error 38 us; skew 500 ppm
I20260924 00:54:42.039476 20507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:42 dist-test-slave-rtvz krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:42.042897 20507 init.cc:377] Logged in from keytab as kudu/127.0.222.1@KRBTEST.COM (short username kudu)
I20260924 00:54:42.044425 20507 webserver.cc:460] Webserver started at http://127.0.222.1:41689/ using document root <none> and password file <none>
I20260924 00:54:42.045262 20507 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:42.045411 20507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:42.045773 20507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:42.048843 20507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "1a4ca7647f6a4d22a07e898590d5ab3e"
format_stamp: "Formatted at 2026-09-24 00:54:42 on dist-test-slave-rtvz"
server_key: "4d6d7881c9626735cc448525b993ab64"
server_key_iv: "bfaf3fc5eb00f9e1eaaccd311e4b80e9"
server_key_version: "encryptionkey@0"
I20260924 00:54:42.049639 20507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "1a4ca7647f6a4d22a07e898590d5ab3e"
format_stamp: "Formatted at 2026-09-24 00:54:42 on dist-test-slave-rtvz"
server_key: "4d6d7881c9626735cc448525b993ab64"
server_key_iv: "bfaf3fc5eb00f9e1eaaccd311e4b80e9"
server_key_version: "encryptionkey@0"
I20260924 00:54:42.055038 20507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:54:42.058746 20522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:42.060305 20507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:42.060501 20507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "1a4ca7647f6a4d22a07e898590d5ab3e"
format_stamp: "Formatted at 2026-09-24 00:54:42 on dist-test-slave-rtvz"
server_key: "4d6d7881c9626735cc448525b993ab64"
server_key_iv: "bfaf3fc5eb00f9e1eaaccd311e4b80e9"
server_key_version: "encryptionkey@0"
I20260924 00:54:42.060729 20507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:42.091565 20507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:42.095932 20507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:42.096192 20507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:42.097184 20507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:42.098600 20507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:42.098675 20507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:42.098775 20507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:42.098839 20507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:42.130442 20507 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:37251
I20260924 00:54:42.130494 20635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:37251 every 8 connection(s)
I20260924 00:54:42.131721 20507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:54:42.139787 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 20507
I20260924 00:54:42.139956 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:54:42.140393 888 external_mini_cluster.cc:1428] Setting key 674752abe3484d1fe66eaf0f93b9814e
Sep 24 00:54:42 dist-test-slave-rtvz krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.1@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:42.152352 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:42.134524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:46677 (local address 127.0.222.62:46361)
0924 00:54:42.135057 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:42.135062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:42.136097 (+ 1035us) server_negotiation.cc:407] Connection header received
0924 00:54:42.137997 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:42.138004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:42.138109 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:42.138201 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:42.139795 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:42.140658 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:42.142290 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:42.142692 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:42.142804 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:42.146340 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:42.146345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:42.146348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:42.146381 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:42.149124 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:42.149924 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:42.149935 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:42.149938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:42.150065 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:42.150521 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:42.150525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:42.150527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:42.150815 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:42.150962 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:42.151726 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 00:54:42.151984 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":219,"threads_started":1}
I20260924 00:54:42.153455 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:42.135096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46361 (local address 127.0.222.1:46677)
0924 00:54:42.135872 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:42.135912 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:42.137720 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:42.138456 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:42.138466 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:42.139020 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:42.139620 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:42.139639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:42.140795 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:42.140798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:42.142154 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:42.142167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:42.142924 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:42.142932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:42.143174 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:42.143838 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:42.143860 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:42.146126 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:42.149296 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:42.149307 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:42.149325 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:42.149784 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:42.150173 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:42.150178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:42.150182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:42.150384 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:42.150945 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:42.150954 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:42.151330 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 00:54:42.151584 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 00:54:42.152183 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":295,"threads_started":1}
I20260924 00:54:42.155274 20636 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46361
I20260924 00:54:42.155638 20636 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:42.156489 20636 heartbeater.cc:511] Master 127.0.222.62:46361 requested a full tablet report, sending...
I20260924 00:54:42.158798 20446 ts_manager.cc:194] Registered new tserver with Master: 1a4ca7647f6a4d22a07e898590d5ab3e (127.0.222.1:37251)
I20260924 00:54:42.160388 20446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.1@KRBTEST.COM'} at 127.0.222.1:46677
WARNING: no policy specified for kudu/127.0.222.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
Entry for principal kudu/127.0.222.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab.
I20260924 00:54:42.194963 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46361
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:42.529879 20642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:42.530165 20642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:42.530260 20642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:42.537524 20642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:42.537642 20642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:42.537763 20642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:54:42.547879 20642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.2.keytab
--principal=kudu/127.0.222.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:42.549310 20642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:42.550621 20642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:42.564116 20648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:42.564124 20647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:42.564648 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:42.565049 20642 server_base.cc:1034] running on GCE node
I20260924 00:54:42.565791 20642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:42.566879 20642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:42.568120 20642 hybrid_clock.cc:648] HybridClock initialized: now 1790211282568048 us; error 52 us; skew 500 ppm
I20260924 00:54:42.568477 20642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:42 dist-test-slave-rtvz krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:42.572214 20642 init.cc:377] Logged in from keytab as kudu/127.0.222.2@KRBTEST.COM (short username kudu)
I20260924 00:54:42.573944 20642 webserver.cc:460] Webserver started at http://127.0.222.2:45027/ using document root <none> and password file <none>
I20260924 00:54:42.574680 20642 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:42.574816 20642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:42.575160 20642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:42.578516 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "449b11c51e764dbaa3f846789b616c1c"
format_stamp: "Formatted at 2026-09-24 00:54:42 on dist-test-slave-rtvz"
server_key: "161a4cb19bde414121f31e320e5f6712"
server_key_iv: "c2aae30ef92e89388cb0a26febe4d330"
server_key_version: "encryptionkey@0"
I20260924 00:54:42.579424 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "449b11c51e764dbaa3f846789b616c1c"
format_stamp: "Formatted at 2026-09-24 00:54:42 on dist-test-slave-rtvz"
server_key: "161a4cb19bde414121f31e320e5f6712"
server_key_iv: "c2aae30ef92e89388cb0a26febe4d330"
server_key_version: "encryptionkey@0"
I20260924 00:54:42.585127 20642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:54:42.588970 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:42.590499 20642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:42.590654 20642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "449b11c51e764dbaa3f846789b616c1c"
format_stamp: "Formatted at 2026-09-24 00:54:42 on dist-test-slave-rtvz"
server_key: "161a4cb19bde414121f31e320e5f6712"
server_key_iv: "c2aae30ef92e89388cb0a26febe4d330"
server_key_version: "encryptionkey@0"
I20260924 00:54:42.590875 20642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:42.605140 20642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:42.609714 20642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:42.609972 20642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:42.610962 20642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:42.612327 20642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:42.612402 20642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:42.612488 20642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:42.612541 20642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:42.644135 20642 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:46169
I20260924 00:54:42.644178 20770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:46169 every 8 connection(s)
I20260924 00:54:42.645587 20642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:54:42.649924 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 20642
I20260924 00:54:42.650195 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:54:42.650758 888 external_mini_cluster.cc:1428] Setting key 3c30669bb1f46b6b0bd9341824754d38
Sep 24 00:54:42 dist-test-slave-rtvz krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.2@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:42.666806 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:42.648323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:38935 (local address 127.0.222.62:46361)
0924 00:54:42.648487 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:42.648492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:42.649875 (+ 1383us) server_negotiation.cc:407] Connection header received
0924 00:54:42.652262 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:42.652268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:42.652351 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:42.652438 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:42.654211 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:42.654909 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:42.656496 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:42.656916 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:42.657036 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:42.660721 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:42.660726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:42.660730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:42.660763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:42.663275 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:42.664085 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:42.664090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:42.664092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:42.664228 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:42.664687 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:42.664692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:42.664695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:42.665018 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:42.665148 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:42.666170 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0924 00:54:42.666441 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:54:42.667853 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:42.648818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46361 (local address 127.0.222.2:38935)
0924 00:54:42.649680 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:42.649740 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:42.651994 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:42.652771 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:42.652784 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:42.653414 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:42.654038 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:42.654057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:42.655068 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:42.655072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:42.656349 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:42.656361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:42.657145 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:42.657153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:42.657423 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:42.658062 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:42.658086 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:42.660459 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:42.663426 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:42.663436 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:42.663455 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:42.663950 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:42.664334 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:42.664339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:42.664343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:42.664547 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:42.665141 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:42.665150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:42.665646 (+ 496us) client_negotiation.cc:769] Sending connection context
0924 00:54:42.666045 (+ 399us) client_negotiation.cc:240] Negotiation successful
0924 00:54:42.666626 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":324,"threads_started":1}
I20260924 00:54:42.669480 20771 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46361
I20260924 00:54:42.669788 20771 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:42.670511 20771 heartbeater.cc:511] Master 127.0.222.62:46361 requested a full tablet report, sending...
I20260924 00:54:42.671963 20446 ts_manager.cc:194] Registered new tserver with Master: 449b11c51e764dbaa3f846789b616c1c (127.0.222.2:46169)
I20260924 00:54:42.672835 20446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.2@KRBTEST.COM'} at 127.0.222.2:38935
WARNING: no policy specified for kudu/127.0.222.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
Entry for principal kudu/127.0.222.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab.
I20260924 00:54:42.708153 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46361
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:34367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:43.042956 20776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:43.043272 20776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:43.043406 20776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:43.050866 20776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:43.051005 20776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:43.051155 20776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:54:43.061178 20776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:34367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/krb5kdc/kudu_127.0.222.3.keytab
--principal=kudu/127.0.222.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:46361
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:43.062566 20776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:43.063952 20776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:43.077581 20781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:43.077777 20782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:43.077612 20784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:43.079530 20776 server_base.cc:1034] running on GCE node
I20260924 00:54:43.080196 20776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:43.081188 20776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:43.082449 20776 hybrid_clock.cc:648] HybridClock initialized: now 1790211283082389 us; error 45 us; skew 500 ppm
I20260924 00:54:43.082778 20776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:43 dist-test-slave-rtvz krb5kdc[20400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:43.086041 20776 init.cc:377] Logged in from keytab as kudu/127.0.222.3@KRBTEST.COM (short username kudu)
I20260924 00:54:43.087560 20776 webserver.cc:460] Webserver started at http://127.0.222.3:45995/ using document root <none> and password file <none>
I20260924 00:54:43.088270 20776 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:43.088376 20776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:43.088685 20776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:43.091706 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "3db0468194004595859e843529121d35"
format_stamp: "Formatted at 2026-09-24 00:54:43 on dist-test-slave-rtvz"
server_key: "160640c16b0035a95c6aa705714ced46"
server_key_iv: "bca867bf0695f6814efec22fc05fcf6b"
server_key_version: "encryptionkey@0"
I20260924 00:54:43.092504 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "3db0468194004595859e843529121d35"
format_stamp: "Formatted at 2026-09-24 00:54:43 on dist-test-slave-rtvz"
server_key: "160640c16b0035a95c6aa705714ced46"
server_key_iv: "bca867bf0695f6814efec22fc05fcf6b"
server_key_version: "encryptionkey@0"
I20260924 00:54:43.098234 20776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 00:54:43.102025 20791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:43.103534 20776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:43.103719 20776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "3db0468194004595859e843529121d35"
format_stamp: "Formatted at 2026-09-24 00:54:43 on dist-test-slave-rtvz"
server_key: "160640c16b0035a95c6aa705714ced46"
server_key_iv: "bca867bf0695f6814efec22fc05fcf6b"
server_key_version: "encryptionkey@0"
I20260924 00:54:43.103919 20776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:43.118534 20776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:43.122713 20776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:43.122969 20776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:43.123893 20776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:43.125268 20776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:43.125355 20776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:43.125494 20776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:43.125556 20776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:43.156172 20776 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:33999
I20260924 00:54:43.156207 20904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:33999 every 8 connection(s)
I20260924 00:54:43.157511 20776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:54:43.162683 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 20776
I20260924 00:54:43.162876 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:54:43.163399 888 external_mini_cluster.cc:1428] Setting key 3c2c6aeb412a1f8376408d2f5b66c76c
I20260924 00:54:43.163601 20636 heartbeater.cc:503] Master 127.0.222.62:46361 was elected leader, sending a full tablet report...
Sep 24 00:54:43 dist-test-slave-rtvz krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.3@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:54:43.179107 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.160186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:41035 (local address 127.0.222.62:46361)
0924 00:54:43.160394 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.160400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.161620 (+ 1220us) server_negotiation.cc:407] Connection header received
0924 00:54:43.163977 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.163984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.164070 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.164139 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:43.166514 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.167322 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.168879 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.169353 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.169458 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.172950 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:43.172956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:43.172960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:43.172998 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:43.175789 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.176516 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.176521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.176524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.176664 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.177056 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.177061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.177064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.177478 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:43.177633 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.178477 (+ 844us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.178701 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:54:43.180184 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.160668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46361 (local address 127.0.222.3:41035)
0924 00:54:43.161420 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:43.161510 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:43.163629 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:43.164722 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:43.164734 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:43.165522 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:43.166343 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.166370 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.167454 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.167457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.168743 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.168755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.169561 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.169566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.169771 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.170410 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:54:43.170438 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:43.172653 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:43.175939 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:43.175947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:43.175962 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:43.176392 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:43.176759 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:54:43.176762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:54:43.176764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:54:43.176948 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:54:43.177616 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:43.177624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:43.178077 (+ 453us) client_negotiation.cc:769] Sending connection context
0924 00:54:43.178353 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 00:54:43.178867 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":279,"threads_started":1}
I20260924 00:54:43.181644 20905 heartbeater.cc:347] Connected to a master server at 127.0.222.62:46361
I20260924 00:54:43.181936 20905 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:43.182667 20905 heartbeater.cc:511] Master 127.0.222.62:46361 requested a full tablet report, sending...
I20260924 00:54:43.184072 20445 ts_manager.cc:194] Registered new tserver with Master: 3db0468194004595859e843529121d35 (127.0.222.3:33999)
I20260924 00:54:43.184870 20445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.222.3@KRBTEST.COM'} at 127.0.222.3:41035
I20260924 00:54:43.191632 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:54:43.209578 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.196257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.0.222.62:46361)
0924 00:54:43.196421 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.196425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.196689 (+ 264us) server_negotiation.cc:407] Connection header received
0924 00:54:43.196838 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.196842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.196911 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.197079 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:43.198667 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.199530 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.200905 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.201394 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.201496 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.203979 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:43.203985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:43.203988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:43.204024 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:43.206728 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.207474 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.207481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.207484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.207663 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.208080 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.208085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.208088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.208493 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:43.208696 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.209055 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.209289 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:54:43.214035 20445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:54:43.221020 20445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:54:43.265798 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.236323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.0.222.1:37251)
0924 00:54:43.256872 (+ 20549us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.256883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.257498 (+ 615us) server_negotiation.cc:407] Connection header received
0924 00:54:43.257807 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.257816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.258149 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.258333 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:43.259141 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.260722 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.263167 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.264268 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.264389 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.264633 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.265274 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.265458 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20365,"thread_start_us":20214,"threads_started":1}
I20260924 00:54:43.266822 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.234090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.0.222.3:33999)
0924 00:54:43.259060 (+ 24970us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.259068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.259101 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:54:43.259189 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.259197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.259435 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.259622 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:43.260246 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.261713 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.264157 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.265761 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.265885 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.266134 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.266338 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.266498 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24781,"thread_start_us":24616,"threads_started":1}
I20260924 00:54:43.266904 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.233728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:33999 (local address 127.0.0.1:35372)
0924 00:54:43.256782 (+ 23054us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:43.256805 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:43.256955 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:43.259629 (+ 2674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:43.259634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:43.259657 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:43.260063 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.260077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.261831 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.261836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.263987 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.264003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.265943 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.265949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.266111 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.266141 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 00:54:43.266222 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:54:43.266442 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22892,"mutex_wait_us":29,"thread_start_us":22434,"threads_started":1}
I20260924 00:54:43.266942 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.236133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:37251 (local address 127.0.0.1:38834)
0924 00:54:43.257345 (+ 21212us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:43.257361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:43.257608 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:43.258519 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:43.258527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:43.258558 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:43.258933 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.258946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.260883 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.260888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.263016 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.263028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.264763 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.264770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.264973 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.265042 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 00:54:43.266259 (+ 1217us) client_negotiation.cc:240] Negotiation successful
0924 00:54:43.266458 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21094,"mutex_wait_us":31,"thread_start_us":18160,"threads_started":1}
I20260924 00:54:43.267488 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.233624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.0.222.2:46169)
0924 00:54:43.259347 (+ 25723us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.259354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.259377 (+ 23us) server_negotiation.cc:407] Connection header received
0924 00:54:43.259449 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.259454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.259632 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.259820 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:43.260531 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.262068 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.264556 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.265914 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.266044 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.266232 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.267001 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.267178 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25586,"thread_start_us":25479,"threads_started":1}
I20260924 00:54:43.267923 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.233431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:46169 (local address 127.0.0.1:50658)
0924 00:54:43.254854 (+ 21423us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:43.254911 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:43.255175 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:43.259936 (+ 4761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:43.259941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:43.259967 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:43.260358 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.260368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.262205 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.262209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.264375 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.264389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.266377 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.266383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.266540 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.266575 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 00:54:43.266660 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:54:43.266780 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21055,"mutex_wait_us":23,"thread_start_us":23628,"threads_started":1}
I20260924 00:54:43.270931 20840 tablet_service.cc:1467] Processing CreateTablet for tablet 816e890f7c3d404ea9c4f27ea8bdd38f (DEFAULT_TABLE table=test-table [id=1e4f775f03b045f784d0c217d45cf400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:43.271176 20706 tablet_service.cc:1467] Processing CreateTablet for tablet 816e890f7c3d404ea9c4f27ea8bdd38f (DEFAULT_TABLE table=test-table [id=1e4f775f03b045f784d0c217d45cf400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:43.271104 20571 tablet_service.cc:1467] Processing CreateTablet for tablet 816e890f7c3d404ea9c4f27ea8bdd38f (DEFAULT_TABLE table=test-table [id=1e4f775f03b045f784d0c217d45cf400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:43.272495 20840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e890f7c3d404ea9c4f27ea8bdd38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:43.272531 20706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e890f7c3d404ea9c4f27ea8bdd38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:43.272540 20571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e890f7c3d404ea9c4f27ea8bdd38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:43.283948 20923 tablet_bootstrap.cc:492] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: Bootstrap starting.
I20260924 00:54:43.286582 20924 tablet_bootstrap.cc:492] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Bootstrap starting.
I20260924 00:54:43.287408 20925 tablet_bootstrap.cc:492] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: Bootstrap starting.
I20260924 00:54:43.289033 20923 tablet_bootstrap.cc:654] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:43.290297 20924 tablet_bootstrap.cc:654] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:43.290356 20923 log.cc:824] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:43.290753 20925 tablet_bootstrap.cc:654] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:43.291384 20924 log.cc:824] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:43.291770 20925 log.cc:824] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:43.293743 20923 tablet_bootstrap.cc:492] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: No bootstrap required, opened a new log
I20260924 00:54:43.294035 20923 ts_tablet_manager.cc:1397] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 00:54:43.294401 20924 tablet_bootstrap.cc:492] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: No bootstrap required, opened a new log
I20260924 00:54:43.294560 20925 tablet_bootstrap.cc:492] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: No bootstrap required, opened a new log
I20260924 00:54:43.294667 20924 ts_tablet_manager.cc:1397] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 00:54:43.294803 20925 ts_tablet_manager.cc:1397] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 00:54:43.302261 20925 raft_consensus.cc:348] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.302672 20925 raft_consensus.cc:374] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:43.302766 20925 raft_consensus.cc:729] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449b11c51e764dbaa3f846789b616c1c, State: Initialized, Role: FOLLOWER
I20260924 00:54:43.303359 20925 consensus_queue.cc:260] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.304646 20771 heartbeater.cc:503] Master 127.0.222.62:46361 was elected leader, sending a full tablet report...
I20260924 00:54:43.304513 20923 raft_consensus.cc:348] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.304950 20923 raft_consensus.cc:374] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:43.305051 20923 raft_consensus.cc:729] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db0468194004595859e843529121d35, State: Initialized, Role: FOLLOWER
I20260924 00:54:43.305249 20925 ts_tablet_manager.cc:1428] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260924 00:54:43.305653 20924 raft_consensus.cc:348] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.305739 20923 consensus_queue.cc:260] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.305979 20924 raft_consensus.cc:374] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:43.306084 20924 raft_consensus.cc:729] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4ca7647f6a4d22a07e898590d5ab3e, State: Initialized, Role: FOLLOWER
I20260924 00:54:43.306595 20924 consensus_queue.cc:260] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.312853 20905 heartbeater.cc:503] Master 127.0.222.62:46361 was elected leader, sending a full tablet report...
I20260924 00:54:43.313944 20923 ts_tablet_manager.cc:1428] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260924 00:54:43.314618 20924 ts_tablet_manager.cc:1428] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260924 00:54:43.323906 20930 raft_consensus.cc:482] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:43.324201 20930 raft_consensus.cc:504] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.326188 20930 leader_election.cc:288] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 449b11c51e764dbaa3f846789b616c1c (127.0.222.2:46169), 1a4ca7647f6a4d22a07e898590d5ab3e (127.0.222.1:37251)
I20260924 00:54:43.333128 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.326590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:42701 (local address 127.0.222.2:46169)
0924 00:54:43.327081 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.327088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.327110 (+ 22us) server_negotiation.cc:407] Connection header received
0924 00:54:43.327201 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.327208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.327298 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.327426 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:43.327780 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.328798 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.331074 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.332217 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.332319 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.332395 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.332765 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.332866 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
I20260924 00:54:43.333190 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.326627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:46169 (local address 127.0.222.3:42701)
0924 00:54:43.326843 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:43.326867 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:43.326992 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:43.327449 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:43.327453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:43.327474 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:43.327629 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.327636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.328942 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.328948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.330840 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.330854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.332388 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.332394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.332606 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.332644 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:54:43.332751 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 00:54:43.332894 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 00:54:43.334221 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.326746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:43513 (local address 127.0.222.1:37251)
0924 00:54:43.326954 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.326960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.327715 (+ 755us) server_negotiation.cc:407] Connection header received
0924 00:54:43.327857 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.327860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.327920 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.328041 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:43.328736 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.329976 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.332134 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.333312 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.333443 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.333546 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.333885 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.333987 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:54:43.334336 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.326643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:37251 (local address 127.0.222.3:43513)
0924 00:54:43.327609 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:43.327624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:43.327745 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:43.328079 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:43.328085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:43.328112 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:43.328513 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.328526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.330145 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.330149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.331923 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:43.331938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.333571 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.333577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:43.333730 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.333754 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 00:54:43.333835 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:54:43.333976 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":61,"thread_start_us":491,"threads_started":1}
I20260924 00:54:43.334398 20726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e890f7c3d404ea9c4f27ea8bdd38f" candidate_uuid: "3db0468194004595859e843529121d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449b11c51e764dbaa3f846789b616c1c" is_pre_election: true
I20260924 00:54:43.334986 20726 raft_consensus.cc:2454] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db0468194004595859e843529121d35 in term 0.
I20260924 00:54:43.335314 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e890f7c3d404ea9c4f27ea8bdd38f" candidate_uuid: "3db0468194004595859e843529121d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" is_pre_election: true
I20260924 00:54:43.335709 20794 leader_election.cc:302] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db0468194004595859e843529121d35, 449b11c51e764dbaa3f846789b616c1c; no voters:
I20260924 00:54:43.335805 20591 raft_consensus.cc:2454] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db0468194004595859e843529121d35 in term 0.
I20260924 00:54:43.336090 20930 raft_consensus.cc:2781] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:54:43.336238 20930 raft_consensus.cc:482] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:54:43.336369 20930 raft_consensus.cc:3037] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:43.338807 20930 raft_consensus.cc:504] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.339376 20930 leader_election.cc:288] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [CANDIDATE]: Term 1 election: Requested vote from peers 449b11c51e764dbaa3f846789b616c1c (127.0.222.2:46169), 1a4ca7647f6a4d22a07e898590d5ab3e (127.0.222.1:37251)
I20260924 00:54:43.339929 20726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e890f7c3d404ea9c4f27ea8bdd38f" candidate_uuid: "3db0468194004595859e843529121d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449b11c51e764dbaa3f846789b616c1c"
I20260924 00:54:43.340029 20591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e890f7c3d404ea9c4f27ea8bdd38f" candidate_uuid: "3db0468194004595859e843529121d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e"
I20260924 00:54:43.340158 20726 raft_consensus.cc:3037] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:43.340262 20591 raft_consensus.cc:3037] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:43.343426 20591 raft_consensus.cc:2454] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db0468194004595859e843529121d35 in term 1.
I20260924 00:54:43.343434 20726 raft_consensus.cc:2454] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db0468194004595859e843529121d35 in term 1.
I20260924 00:54:43.343864 20794 leader_election.cc:302] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db0468194004595859e843529121d35, 449b11c51e764dbaa3f846789b616c1c; no voters:
I20260924 00:54:43.344192 20930 raft_consensus.cc:2781] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:43.344621 20930 raft_consensus.cc:686] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 1 LEADER]: Becoming Leader. State: Replica: 3db0468194004595859e843529121d35, State: Running, Role: LEADER
I20260924 00:54:43.345177 20930 consensus_queue.cc:237] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } }
I20260924 00:54:43.350390 20445 catalog_manager.cc:5495] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db0468194004595859e843529121d35 (127.0.222.3). New cstate: current_term: 1 leader_uuid: "3db0468194004595859e843529121d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3db0468194004595859e843529121d35" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 33999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 } health_report { overall_health: UNKNOWN } } }
I20260924 00:54:43.368912 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.363424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.0.222.3:33999)
0924 00:54:43.363587 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.363591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.363607 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:54:43.363704 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.363708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.363797 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.364006 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:54:43.364413 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.365346 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.366768 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.367308 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.367423 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.367891 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:54:43.368418 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:54:43.368571 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.368643 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.368714 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:54:43.376839 20726 raft_consensus.cc:1261] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 1 FOLLOWER]: Refusing update from remote peer 3db0468194004595859e843529121d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:43.376842 20591 raft_consensus.cc:1261] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 1 FOLLOWER]: Refusing update from remote peer 3db0468194004595859e843529121d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:43.377802 20933 consensus_queue.cc:1035] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a4ca7647f6a4d22a07e898590d5ab3e" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:43.378129 20930 consensus_queue.cc:1035] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449b11c51e764dbaa3f846789b616c1c" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 46169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:54:43.384080 20637 tablet.cc:2369] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:54:43.390533 20936 mvcc.cc:205] Tried to move back new op lower bound from 7332705416702271488 to 7332705416586006528. Current Snapshot: MvccSnapshot[applied={T|T < 7332705416702271488}]
I20260924 00:54:43.391795 20938 mvcc.cc:205] Tried to move back new op lower bound from 7332705416702271488 to 7332705416586006528. Current Snapshot: MvccSnapshot[applied={T|T < 7332705416702271488}]
I20260924 00:54:43.405061 20446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:43.405340 20446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:43.409001 20446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38ef7b085748468bbef138bb2cfb7612: Sending DeleteTablet for 3 replicas of tablet 816e890f7c3d404ea9c4f27ea8bdd38f
I20260924 00:54:43.410200 20840 tablet_service.cc:1514] Processing DeleteTablet for tablet 816e890f7c3d404ea9c4f27ea8bdd38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:43 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:35372
I20260924 00:54:43.410190 20706 tablet_service.cc:1514] Processing DeleteTablet for tablet 816e890f7c3d404ea9c4f27ea8bdd38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:43 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:50658
I20260924 00:54:43.411149 20947 tablet_replica.cc:330] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: stopping tablet replica
I20260924 00:54:43.411645 20947 raft_consensus.cc:2229] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:54:43.411602 20571 tablet_service.cc:1514] Processing DeleteTablet for tablet 816e890f7c3d404ea9c4f27ea8bdd38f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:43 UTC) from {username='kudu', principal='kudu/127.0.222.62@KRBTEST.COM'} at 127.0.0.1:38834
I20260924 00:54:43.412554 20946 tablet_replica.cc:330] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: stopping tablet replica
I20260924 00:54:43.413021 20946 raft_consensus.cc:2229] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:43.413650 20946 raft_consensus.cc:2258] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:43.413928 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 20507
I20260924 00:54:43.414194 20947 raft_consensus.cc:2258] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:43.416877 20947 ts_tablet_manager.cc:1905] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:43.424211 20948 tablet_replica.cc:330] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: stopping tablet replica
I20260924 00:54:43.424780 20948 raft_consensus.cc:2229] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:43.425395 20948 raft_consensus.cc:2258] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:43.426954 20946 ts_tablet_manager.cc:1905] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:43.427516 20948 ts_tablet_manager.cc:1905] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 00:54:43 dist-test-slave-rtvz krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.1@KRBTEST.COM
I20260924 00:54:43.432783 20947 ts_tablet_manager.cc:1918] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:43.433115 20947 log.cc:1196] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/816e890f7c3d404ea9c4f27ea8bdd38f
I20260924 00:54:43.433822 20947 ts_tablet_manager.cc:1939] T 816e890f7c3d404ea9c4f27ea8bdd38f P 3db0468194004595859e843529121d35: Deleting consensus metadata
I20260924 00:54:43.434664 20948 ts_tablet_manager.cc:1918] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:43.434837 20948 log.cc:1196] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-0/wal/wals/816e890f7c3d404ea9c4f27ea8bdd38f
I20260924 00:54:43.435451 20948 ts_tablet_manager.cc:1939] T 816e890f7c3d404ea9c4f27ea8bdd38f P 1a4ca7647f6a4d22a07e898590d5ab3e: Deleting consensus metadata
I20260924 00:54:43.436820 20946 ts_tablet_manager.cc:1918] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:43.437635 20431 catalog_manager.cc:4841] TS 1a4ca7647f6a4d22a07e898590d5ab3e (127.0.222.1:37251): tablet 816e890f7c3d404ea9c4f27ea8bdd38f (table test-table [id=1e4f775f03b045f784d0c217d45cf400]) successfully deleted
I20260924 00:54:43.438277 20946 log.cc:1196] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/816e890f7c3d404ea9c4f27ea8bdd38f
I20260924 00:54:43.438329 20433 catalog_manager.cc:4841] TS 3db0468194004595859e843529121d35 (127.0.222.3:33999): tablet 816e890f7c3d404ea9c4f27ea8bdd38f (table test-table [id=1e4f775f03b045f784d0c217d45cf400]) successfully deleted
I20260924 00:54:43.438964 20946 ts_tablet_manager.cc:1939] T 816e890f7c3d404ea9c4f27ea8bdd38f P 449b11c51e764dbaa3f846789b616c1c: Deleting consensus metadata
I20260924 00:54:43.441038 20432 catalog_manager.cc:4841] TS 449b11c51e764dbaa3f846789b616c1c (127.0.222.2:46169): tablet 816e890f7c3d404ea9c4f27ea8bdd38f (table test-table [id=1e4f775f03b045f784d0c217d45cf400]) successfully deleted
I20260924 00:54:43.441238 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.415252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.0.222.1:37251)
0924 00:54:43.415456 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.415462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.422060 (+ 6598us) server_negotiation.cc:407] Connection header received
0924 00:54:43.422255 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.422262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.422362 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.422458 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:43.424360 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.425529 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.427376 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.428012 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.428114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.434152 (+ 6038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:43.434170 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:43.434189 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:43.434243 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:43.438161 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.439057 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.439066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.439070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.439220 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.439616 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.439622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.439625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.440027 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:43.440305 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.440627 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.440852 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 00:54:43.442329 20631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:43.533474 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 20507
I20260924 00:54:43.548872 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 20642
Sep 24 00:54:43 dist-test-slave-rtvz krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.2@KRBTEST.COM
I20260924 00:54:43.566386 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.549682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.0.222.2:46169)
0924 00:54:43.549853 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.549857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.549875 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:54:43.550157 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.550162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.550252 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.550355 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:43.552004 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.552748 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.554696 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.555203 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.555318 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.560574 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:43.560594 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:43.560611 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:43.560653 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:43.563568 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.564307 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.564315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.564317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.564447 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.564808 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.564812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.564814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.565248 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:43.565526 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.565834 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.566056 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 00:54:43.567183 20766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:43.655512 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 20642
I20260924 00:54:43.675338 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 20776
Sep 24 00:54:43 dist-test-slave-rtvz krb5kdc[20400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.3@KRBTEST.COM
I20260924 00:54:43.691332 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:43.676120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.222.3:33999)
0924 00:54:43.676284 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:43.676290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:43.676360 (+ 70us) server_negotiation.cc:407] Connection header received
0924 00:54:43.676571 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:43.676576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:43.676648 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:43.676737 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:43.678284 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.679080 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.680393 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:43.680867 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:43.680982 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:43.685336 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:43.685351 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:43.685368 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:43.685411 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:43.688370 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.689180 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.689188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.689190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.689375 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:43.689756 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:43.689760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:43.689762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:43.690136 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:43.690466 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:43.690744 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 00:54:43.690962 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 00:54:43.692107 20900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:43.813583 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 20776
I20260924 00:54:43.830355 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 20413
I20260924 00:54:43.831614 20476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:43.942615 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 20413
2026-09-24T00:54:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4197 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T00:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:43Z Disabled control of system clock
I20260924 00:54:44.016733 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34301
--webserver_interface=127.0.222.62
--webserver_port=0
--builtin_ntp_servers=127.0.222.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:34301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:54:44.371115 20957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:44.371418 20957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:44.371500 20957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:44.378911 20957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:44.379040 20957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:44.379077 20957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:44.379132 20957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:44.379174 20957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:44.388566 20957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:34301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:44.389956 20957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:44.391768 20957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:44.403746 20962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:44.403805 20963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:44.403822 20965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:44.404377 20957 server_base.cc:1034] running on GCE node
I20260924 00:54:44.405141 20957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:44.406610 20957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:44.407850 20957 hybrid_clock.cc:648] HybridClock initialized: now 1790211284407793 us; error 48 us; skew 500 ppm
I20260924 00:54:44.408274 20957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:44.410634 20957 webserver.cc:460] Webserver started at http://127.0.222.62:34371/ using document root <none> and password file <none>
I20260924 00:54:44.411332 20957 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:44.411434 20957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:44.411744 20957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:44.414692 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "46019f6701ef4934b95a773141330b1e"
format_stamp: "Formatted at 2026-09-24 00:54:44 on dist-test-slave-rtvz"
server_key: "6b1d9a4e660c153fd146e325a6f0749e"
server_key_iv: "b61931b690c981accfb37f202b349b30"
server_key_version: "encryptionkey@0"
I20260924 00:54:44.415439 20957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "46019f6701ef4934b95a773141330b1e"
format_stamp: "Formatted at 2026-09-24 00:54:44 on dist-test-slave-rtvz"
server_key: "6b1d9a4e660c153fd146e325a6f0749e"
server_key_iv: "b61931b690c981accfb37f202b349b30"
server_key_version: "encryptionkey@0"
I20260924 00:54:44.420606 20957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 00:54:44.424253 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:44.425850 20957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:44.426034 20957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "46019f6701ef4934b95a773141330b1e"
format_stamp: "Formatted at 2026-09-24 00:54:44 on dist-test-slave-rtvz"
server_key: "6b1d9a4e660c153fd146e325a6f0749e"
server_key_iv: "b61931b690c981accfb37f202b349b30"
server_key_version: "encryptionkey@0"
I20260924 00:54:44.426246 20957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:44.440830 20957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:44.441781 20957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:44.442071 20957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:44.459187 20957 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:34301
I20260924 00:54:44.459213 21022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:34301 every 8 connection(s)
I20260924 00:54:44.460565 20957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:44.462984 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 20957
I20260924 00:54:44.463171 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:44.463694 888 external_mini_cluster.cc:1428] Setting key 4137b0644c263f15fb6cc90f8cda5eb4
I20260924 00:54:44.464658 21023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:44.478322 21023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e: Bootstrap starting.
I20260924 00:54:44.478765 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:44.465676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.0.222.62:34301)
0924 00:54:44.466708 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:44.466719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:44.466764 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:54:44.469133 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:44.469157 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:44.469733 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:44.470088 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:44.470786 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:44.472373 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:44.474228 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:44.474952 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:44.475095 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:44.476300 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:44.476307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:44.476322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:44.476861 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:44.476992 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:44.477017 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:44.477102 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:44.477363 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 00:54:44.477703 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":361,"threads_started":1}
I20260924 00:54:44.481781 21023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:44.482779 21023 log.cc:824] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:44.485636 21023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e: No bootstrap required, opened a new log
I20260924 00:54:44.492647 21023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019f6701ef4934b95a773141330b1e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34301 } }
I20260924 00:54:44.493072 21023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:44.493175 21023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46019f6701ef4934b95a773141330b1e, State: Initialized, Role: FOLLOWER
I20260924 00:54:44.493834 21023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019f6701ef4934b95a773141330b1e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34301 } }
I20260924 00:54:44.494055 21023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:44.494177 21023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:44.494318 21023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:44.496410 21023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019f6701ef4934b95a773141330b1e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34301 } }
I20260924 00:54:44.496914 21023 leader_election.cc:302] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46019f6701ef4934b95a773141330b1e; no voters:
I20260924 00:54:44.497575 21023 leader_election.cc:288] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:44.497700 21028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:44.498354 21028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [term 1 LEADER]: Becoming Leader. State: Replica: 46019f6701ef4934b95a773141330b1e, State: Running, Role: LEADER
I20260924 00:54:44.498862 21028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019f6701ef4934b95a773141330b1e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34301 } }
I20260924 00:54:44.499519 21023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:44.501726 21029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46019f6701ef4934b95a773141330b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019f6701ef4934b95a773141330b1e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34301 } } }
I20260924 00:54:44.501752 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46019f6701ef4934b95a773141330b1e. Latest consensus state: current_term: 1 leader_uuid: "46019f6701ef4934b95a773141330b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019f6701ef4934b95a773141330b1e" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34301 } } }
I20260924 00:54:44.502187 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:44.502197 21029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:44.504379 21033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:44.510110 21033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:44.520570 21033 catalog_manager.cc:1347] Generated new cluster ID: 92c48b0d2df04dcf89d14159718ab9df
I20260924 00:54:44.520694 21033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:44.551110 21033 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:44.552512 21033 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:44.561465 21033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e: Generated new TSK 0
I20260924 00:54:44.562433 21033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:44.581802 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.1:0
--local_ip_for_outbound_sockets=127.0.222.1
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34301
--builtin_ntp_servers=127.0.222.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:54:44.920185 21047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:44.920483 21047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:44.920596 21047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:44.928179 21047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:44.928311 21047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:44.928467 21047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.1
I20260924 00:54:44.938301 21047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.222.1
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.1
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:44.939656 21047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:44.941601 21047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:44.955888 21052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:44.955853 21055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:44.955924 21053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:44.956584 21047 server_base.cc:1034] running on GCE node
I20260924 00:54:44.957392 21047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:44.958453 21047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:44.959695 21047 hybrid_clock.cc:648] HybridClock initialized: now 1790211284959646 us; error 43 us; skew 500 ppm
I20260924 00:54:44.960081 21047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:44.962519 21047 webserver.cc:460] Webserver started at http://127.0.222.1:39767/ using document root <none> and password file <none>
I20260924 00:54:44.963236 21047 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:44.963335 21047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:44.963658 21047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:44.966935 21047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/data/instance:
uuid: "76fb6f393d6541d3935585c95ce2ffb1"
format_stamp: "Formatted at 2026-09-24 00:54:44 on dist-test-slave-rtvz"
server_key: "04e4b5010bcc1d3383b47cd089b47b5b"
server_key_iv: "adcbbd10445e3a981fa0a958336ceef9"
server_key_version: "encryptionkey@0"
I20260924 00:54:44.967792 21047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/wal/instance:
uuid: "76fb6f393d6541d3935585c95ce2ffb1"
format_stamp: "Formatted at 2026-09-24 00:54:44 on dist-test-slave-rtvz"
server_key: "04e4b5010bcc1d3383b47cd089b47b5b"
server_key_iv: "adcbbd10445e3a981fa0a958336ceef9"
server_key_version: "encryptionkey@0"
I20260924 00:54:44.973285 21047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 00:54:44.976962 21061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:44.978490 21047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 00:54:44.978704 21047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/wal
uuid: "76fb6f393d6541d3935585c95ce2ffb1"
format_stamp: "Formatted at 2026-09-24 00:54:44 on dist-test-slave-rtvz"
server_key: "04e4b5010bcc1d3383b47cd089b47b5b"
server_key_iv: "adcbbd10445e3a981fa0a958336ceef9"
server_key_version: "encryptionkey@0"
I20260924 00:54:44.978928 21047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:44.999876 21047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:45.000797 21047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:45.001129 21047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:45.002283 21047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:45.003793 21047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:45.003882 21047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:45.003964 21047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:45.004017 21047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:45.034812 21047 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.1:36677
I20260924 00:54:45.034889 21173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.1:36677 every 8 connection(s)
I20260924 00:54:45.036177 21047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/data/info.pb
I20260924 00:54:45.040575 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 21047
I20260924 00:54:45.040743 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-0/wal/instance
I20260924 00:54:45.041181 888 external_mini_cluster.cc:1428] Setting key 2ece9f2b21e63719a99e56faa39e5171
I20260924 00:54:45.044131 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.2:0
--local_ip_for_outbound_sockets=127.0.222.2
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34301
--builtin_ntp_servers=127.0.222.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:54:45.049474 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:45.038881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.1:50169 (local address 127.0.222.62:34301)
0924 00:54:45.039356 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:45.039363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:45.040665 (+ 1302us) server_negotiation.cc:407] Connection header received
0924 00:54:45.042743 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:45.042747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:45.042825 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:45.043030 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:45.044172 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.044946 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.046633 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.047037 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.047153 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:45.048585 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:45.048590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:45.048594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:45.048614 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:45.048662 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:45.048675 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:45.048753 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:45.048977 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 00:54:45.049085 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":220,"threads_started":1}
I20260924 00:54:45.050557 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:45.039428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34301 (local address 127.0.222.1:50169)
0924 00:54:45.040480 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:45.040519 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:45.042423 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:45.043233 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:45.043248 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:45.043366 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:45.043989 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:45.044008 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.045109 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.045115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:45.046473 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:45.046486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.047277 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.047285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:45.047542 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:45.048229 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:45.048232 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:45.048277 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:45.048371 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:45.048769 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:45.048781 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:45.048790 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:54:45.048953 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 00:54:45.049444 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":345,"threads_started":1}
I20260924 00:54:45.052229 21174 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34301
I20260924 00:54:45.052567 21174 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:45.053516 21174 heartbeater.cc:511] Master 127.0.222.62:34301 requested a full tablet report, sending...
I20260924 00:54:45.055835 20988 ts_manager.cc:194] Registered new tserver with Master: 76fb6f393d6541d3935585c95ce2ffb1 (127.0.222.1:36677)
I20260924 00:54:45.057811 20988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.1:50169
W20260924 00:54:45.375030 21178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:45.375321 21178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:45.375404 21178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:45.382733 21178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:45.382839 21178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:45.382947 21178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.2
I20260924 00:54:45.392366 21178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.222.2
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.2
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:45.393719 21178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:45.395473 21178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:45.408669 21183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:45.408697 21184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:45.408737 21186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:45.409533 21178 server_base.cc:1034] running on GCE node
I20260924 00:54:45.410236 21178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:45.411237 21178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:45.412459 21178 hybrid_clock.cc:648] HybridClock initialized: now 1790211285412399 us; error 52 us; skew 500 ppm
I20260924 00:54:45.412786 21178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:45.415249 21178 webserver.cc:460] Webserver started at http://127.0.222.2:38557/ using document root <none> and password file <none>
I20260924 00:54:45.415966 21178 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:45.416081 21178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:45.416393 21178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:45.419392 21178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/data/instance:
uuid: "bac709ce86144eb0b6fd01059c876503"
format_stamp: "Formatted at 2026-09-24 00:54:45 on dist-test-slave-rtvz"
server_key: "f07cad73f85df0db1772934210670c63"
server_key_iv: "b49be6001851abafd5ff5dc88c60b664"
server_key_version: "encryptionkey@0"
I20260924 00:54:45.420259 21178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/wal/instance:
uuid: "bac709ce86144eb0b6fd01059c876503"
format_stamp: "Formatted at 2026-09-24 00:54:45 on dist-test-slave-rtvz"
server_key: "f07cad73f85df0db1772934210670c63"
server_key_iv: "b49be6001851abafd5ff5dc88c60b664"
server_key_version: "encryptionkey@0"
I20260924 00:54:45.425778 21178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:54:45.429451 21192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:45.430982 21178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:45.431154 21178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/wal
uuid: "bac709ce86144eb0b6fd01059c876503"
format_stamp: "Formatted at 2026-09-24 00:54:45 on dist-test-slave-rtvz"
server_key: "f07cad73f85df0db1772934210670c63"
server_key_iv: "b49be6001851abafd5ff5dc88c60b664"
server_key_version: "encryptionkey@0"
I20260924 00:54:45.431371 21178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:45.469959 21178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:45.470811 21178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:45.471122 21178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:45.472111 21178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:45.473723 21178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:45.473817 21178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:45.473922 21178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:45.473992 21178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:45.505162 21178 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.2:40047
I20260924 00:54:45.505235 21304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.2:40047 every 8 connection(s)
I20260924 00:54:45.506497 21178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/data/info.pb
I20260924 00:54:45.512140 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 21178
I20260924 00:54:45.512321 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/wal/instance
I20260924 00:54:45.512959 888 external_mini_cluster.cc:1428] Setting key da568759d277daf13d58b9683a4d2649
I20260924 00:54:45.516289 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.222.3:0
--local_ip_for_outbound_sockets=127.0.222.3
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34301
--builtin_ntp_servers=127.0.222.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:54:45.520848 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:45.509134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.2:41681 (local address 127.0.222.62:34301)
0924 00:54:45.509312 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:45.509317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:45.510770 (+ 1453us) server_negotiation.cc:407] Connection header received
0924 00:54:45.513103 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:45.513109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:45.513224 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:45.513325 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:45.514698 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.515537 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.517929 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.518341 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.518445 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:45.520068 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:45.520074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:45.520078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:45.520097 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:45.520134 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:45.520147 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:45.520246 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:45.520518 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 00:54:45.520607 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:54:45.522236 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:45.509694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34301 (local address 127.0.222.2:41681)
0924 00:54:45.510609 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:45.510649 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:45.512723 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:45.513625 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:45.513634 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:45.513746 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:45.514503 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:45.514529 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.515703 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.515709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:45.517769 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:45.517787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.518566 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.518572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:45.518866 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:45.519685 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:45.519689 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:45.519741 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:45.519850 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:45.520260 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:45.520269 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:45.520276 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:54:45.520472 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 00:54:45.520905 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":291,"threads_started":1}
I20260924 00:54:45.523646 21305 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34301
I20260924 00:54:45.523983 21305 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:45.524838 21305 heartbeater.cc:511] Master 127.0.222.62:34301 requested a full tablet report, sending...
I20260924 00:54:45.526511 20988 ts_manager.cc:194] Registered new tserver with Master: bac709ce86144eb0b6fd01059c876503 (127.0.222.2:40047)
I20260924 00:54:45.527311 20988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.2:41681
W20260924 00:54:45.847247 21308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:45.847558 21308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:45.847676 21308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:45.855366 21308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:45.855484 21308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:45.855641 21308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.222.3
I20260924 00:54:45.865145 21308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.222.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.222.3
--webserver_port=0
--tserver_master_addrs=127.0.222.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.222.3
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:45.866499 21308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:45.868280 21308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:45.881464 21313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:45.881417 21316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:45.881503 21314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:45.882277 21308 server_base.cc:1034] running on GCE node
I20260924 00:54:45.882886 21308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:45.883836 21308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:45.885057 21308 hybrid_clock.cc:648] HybridClock initialized: now 1790211285884989 us; error 54 us; skew 500 ppm
I20260924 00:54:45.885396 21308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 00:54:45.887712 21308 webserver.cc:460] Webserver started at http://127.0.222.3:36485/ using document root <none> and password file <none>
I20260924 00:54:45.888401 21308 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:45.888495 21308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:45.888803 21308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:45.891734 21308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/data/instance:
uuid: "961b6d294f9f4330aed4072854228c9f"
format_stamp: "Formatted at 2026-09-24 00:54:45 on dist-test-slave-rtvz"
server_key: "b17391fa74ed86936222416a36c741b0"
server_key_iv: "ceec1930ed14474ea109121c3a6a91b7"
server_key_version: "encryptionkey@0"
I20260924 00:54:45.892537 21308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/wal/instance:
uuid: "961b6d294f9f4330aed4072854228c9f"
format_stamp: "Formatted at 2026-09-24 00:54:45 on dist-test-slave-rtvz"
server_key: "b17391fa74ed86936222416a36c741b0"
server_key_iv: "ceec1930ed14474ea109121c3a6a91b7"
server_key_version: "encryptionkey@0"
I20260924 00:54:45.897680 21308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 00:54:45.901355 21322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:45.902628 21308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:45.902776 21308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/wal
uuid: "961b6d294f9f4330aed4072854228c9f"
format_stamp: "Formatted at 2026-09-24 00:54:45 on dist-test-slave-rtvz"
server_key: "b17391fa74ed86936222416a36c741b0"
server_key_iv: "ceec1930ed14474ea109121c3a6a91b7"
server_key_version: "encryptionkey@0"
I20260924 00:54:45.902977 21308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:45.925288 21308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:45.926291 21308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:45.926596 21308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:45.927620 21308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:54:45.929271 21308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:54:45.929371 21308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:45.929488 21308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:54:45.929569 21308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:45.969154 21308 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.3:41037
I20260924 00:54:45.969285 21434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.3:41037 every 8 connection(s)
I20260924 00:54:45.970563 21308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/data/info.pb
I20260924 00:54:45.973737 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 21308
I20260924 00:54:45.973896 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/wal/instance
I20260924 00:54:45.974330 888 external_mini_cluster.cc:1428] Setting key 9b59bbd05ec7acb948086b401ced6b9a
I20260924 00:54:45.983187 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:45.973142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:58141 (local address 127.0.222.62:34301)
0924 00:54:45.973377 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:45.973384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:45.975001 (+ 1617us) server_negotiation.cc:407] Connection header received
0924 00:54:45.977136 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:45.977141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:45.977253 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:45.977330 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:45.978409 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.979145 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.980695 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.981125 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.981264 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:45.982507 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:45.982512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:45.982515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:45.982535 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:45.982568 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:45.982580 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:45.982654 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:45.982825 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 00:54:45.982915 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 00:54:45.984242 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:45.973691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:34301 (local address 127.0.222.3:58141)
0924 00:54:45.974843 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:45.974890 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:45.976868 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:45.977564 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:45.977573 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:45.977677 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:54:45.978269 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:45.978287 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.979278 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.979282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:45.980561 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:45.980574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.981348 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.981353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:45.981571 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:45.982173 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:54:45.982175 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:54:45.982214 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:54:45.982299 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:54:45.982658 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:54:45.982663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:54:45.982668 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:54:45.982801 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 00:54:45.983212 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":441,"threads_started":1}
I20260924 00:54:45.985682 21435 heartbeater.cc:347] Connected to a master server at 127.0.222.62:34301
I20260924 00:54:45.985991 21435 heartbeater.cc:464] Registering TS with master...
I20260924 00:54:45.986701 21435 heartbeater.cc:511] Master 127.0.222.62:34301 requested a full tablet report, sending...
I20260924 00:54:45.988129 20988 ts_manager.cc:194] Registered new tserver with Master: 961b6d294f9f4330aed4072854228c9f (127.0.222.3:41037)
I20260924 00:54:45.989084 20988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.222.3:58141
I20260924 00:54:45.989728 888 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:54:46.000080 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:45.993832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.0.222.62:34301)
0924 00:54:45.994038 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:45.994043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:45.994380 (+ 337us) server_negotiation.cc:407] Connection header received
0924 00:54:45.994582 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:45.994586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:45.994679 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:45.994766 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:45.995372 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.996152 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.997715 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:45.998206 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:45.998325 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:45.999336 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:45.999340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:45.999375 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:45.999401 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:45.999445 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:45.999461 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:45.999549 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:45.999685 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 00:54:45.999784 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 00:54:46.004577 20988 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:54:46.012127 20988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:54:46.040583 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.025694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.3:41037 (local address 127.0.0.1:43858)
0924 00:54:46.027678 (+ 1984us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.027713 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:46.027953 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:46.030208 (+ 2255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:46.030217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:46.030248 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:46.030645 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.030658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.033867 (+ 3209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.033872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.037809 (+ 3937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.037824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.039765 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.039770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.039913 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.039951 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:54:46.040057 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 00:54:46.040256 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1750,"thread_start_us":1588,"threads_started":1}
I20260924 00:54:46.040640 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.026304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.0.222.3:41037)
0924 00:54:46.027099 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.027110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.027806 (+ 696us) server_negotiation.cc:407] Connection header received
0924 00:54:46.029316 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.029326 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.029626 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.029794 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:46.030860 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.033687 (+ 2827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.038050 (+ 4363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.039329 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.039419 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.039628 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.040134 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.040307 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":478,"threads_started":1}
I20260924 00:54:46.045032 21370 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbda32dbdf24df6b6c45131b6e51870 (DEFAULT_TABLE table=test-table [id=bdd9097d74dc4052b799aab422c1a3f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:46.046661 21370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbda32dbdf24df6b6c45131b6e51870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:46.049921 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.028054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.0.1:59324)
0924 00:54:46.042059 (+ 14005us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.042077 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:46.042181 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:46.043099 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:46.043104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:46.043129 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:46.043307 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.043317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.045167 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.045172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.047311 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.047326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.049096 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.049102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.049330 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.049363 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:54:46.049452 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:54:46.049604 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12803,"thread_start_us":9728,"threads_started":1}
I20260924 00:54:46.049976 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.028494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.0.222.1:36677)
0924 00:54:46.038732 (+ 10238us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.038750 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.042302 (+ 3552us) server_negotiation.cc:407] Connection header received
0924 00:54:46.042414 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.042421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.042674 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.042865 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:46.043476 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.044970 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.047580 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.048936 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.049049 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.049344 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.049518 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.049670 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10070,"thread_start_us":9953,"threads_started":1}
I20260924 00:54:46.052910 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.025721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:40047 (local address 127.0.0.1:42078)
0924 00:54:46.038093 (+ 12372us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.038114 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:46.038233 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:46.046384 (+ 8151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:46.046389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:46.046415 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:46.046777 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.046788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.048832 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.048837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.050855 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.050870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.052165 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.052171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.052338 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.052371 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:54:46.052456 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:54:46.052600 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12221,"thread_start_us":20384,"threads_started":1}
I20260924 00:54:46.052963 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.025793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.222.2:40047)
0924 00:54:46.045765 (+ 19972us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.045774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.045807 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:54:46.045902 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.045910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.046168 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.046960 (+ 792us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:46.047213 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.048683 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.051073 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.051972 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.052100 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.052342 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.052509 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.052665 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19822,"thread_start_us":19731,"threads_started":1}
I20260924 00:54:46.054113 21109 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbda32dbdf24df6b6c45131b6e51870 (DEFAULT_TABLE table=test-table [id=bdd9097d74dc4052b799aab422c1a3f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:46.056274 21109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbda32dbdf24df6b6c45131b6e51870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:46.056474 21240 tablet_service.cc:1467] Processing CreateTablet for tablet 3cbda32dbdf24df6b6c45131b6e51870 (DEFAULT_TABLE table=test-table [id=bdd9097d74dc4052b799aab422c1a3f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:54:46.057580 21240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cbda32dbdf24df6b6c45131b6e51870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:46.061547 21453 tablet_bootstrap.cc:492] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: Bootstrap starting.
I20260924 00:54:46.061646 21174 heartbeater.cc:503] Master 127.0.222.62:34301 was elected leader, sending a full tablet report...
I20260924 00:54:46.065894 21453 tablet_bootstrap.cc:654] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:46.067243 21453 log.cc:824] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:46.070403 21455 tablet_bootstrap.cc:492] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1: Bootstrap starting.
I20260924 00:54:46.070752 21453 tablet_bootstrap.cc:492] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: No bootstrap required, opened a new log
I20260924 00:54:46.071099 21453 ts_tablet_manager.cc:1397] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 00:54:46.072124 21456 tablet_bootstrap.cc:492] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: Bootstrap starting.
I20260924 00:54:46.074661 21455 tablet_bootstrap.cc:654] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:46.075739 21456 tablet_bootstrap.cc:654] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:46.076813 21456 log.cc:824] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:46.076939 21455 log.cc:824] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:46.079826 21455 tablet_bootstrap.cc:492] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1: No bootstrap required, opened a new log
I20260924 00:54:46.080057 21456 tablet_bootstrap.cc:492] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: No bootstrap required, opened a new log
I20260924 00:54:46.080127 21455 ts_tablet_manager.cc:1397] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 00:54:46.080338 21456 ts_tablet_manager.cc:1397] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 00:54:46.081156 21453 raft_consensus.cc:348] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.081687 21453 raft_consensus.cc:374] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:46.081794 21453 raft_consensus.cc:729] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961b6d294f9f4330aed4072854228c9f, State: Initialized, Role: FOLLOWER
I20260924 00:54:46.082465 21453 consensus_queue.cc:260] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.083696 21435 heartbeater.cc:503] Master 127.0.222.62:34301 was elected leader, sending a full tablet report...
I20260924 00:54:46.086251 21453 ts_tablet_manager.cc:1428] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260924 00:54:46.089507 21455 raft_consensus.cc:348] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.089973 21455 raft_consensus.cc:374] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:46.090088 21455 raft_consensus.cc:729] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fb6f393d6541d3935585c95ce2ffb1, State: Initialized, Role: FOLLOWER
I20260924 00:54:46.090185 21456 raft_consensus.cc:348] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.090606 21456 raft_consensus.cc:374] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:46.090706 21456 raft_consensus.cc:729] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac709ce86144eb0b6fd01059c876503, State: Initialized, Role: FOLLOWER
I20260924 00:54:46.090759 21455 consensus_queue.cc:260] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.091333 21456 consensus_queue.cc:260] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.092545 21455 ts_tablet_manager.cc:1428] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 00:54:46.093396 21305 heartbeater.cc:503] Master 127.0.222.62:34301 was elected leader, sending a full tablet report...
I20260924 00:54:46.093698 21459 raft_consensus.cc:482] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:54:46.094111 21456 ts_tablet_manager.cc:1428] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260924 00:54:46.093937 21459 raft_consensus.cc:504] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.095870 21459 leader_election.cc:288] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac709ce86144eb0b6fd01059c876503 (127.0.222.2:40047), 76fb6f393d6541d3935585c95ce2ffb1 (127.0.222.1:36677)
I20260924 00:54:46.102635 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.096073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.2:40047 (local address 127.0.222.3:40283)
0924 00:54:46.096284 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.096311 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:46.096428 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:46.097247 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:46.097253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:46.097281 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:46.097473 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.097486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.098946 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.098951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.100552 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.100567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.101952 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.101958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.102158 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.102197 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 00:54:46.102280 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:54:46.102389 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 00:54:46.102715 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.096575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:40283 (local address 127.0.222.2:40047)
0924 00:54:46.096779 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.096785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.096804 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:54:46.096863 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.096868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.096958 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.097043 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:46.097658 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.098794 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.100724 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.101841 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.101941 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.102005 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.102327 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.102457 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:54:46.103691 21260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbda32dbdf24df6b6c45131b6e51870" candidate_uuid: "961b6d294f9f4330aed4072854228c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac709ce86144eb0b6fd01059c876503" is_pre_election: true
I20260924 00:54:46.104184 21260 raft_consensus.cc:2454] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 961b6d294f9f4330aed4072854228c9f in term 0.
I20260924 00:54:46.104928 21325 leader_election.cc:302] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 961b6d294f9f4330aed4072854228c9f, bac709ce86144eb0b6fd01059c876503; no voters:
I20260924 00:54:46.105366 21459 raft_consensus.cc:2781] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:54:46.105515 21459 raft_consensus.cc:482] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:54:46.105615 21459 raft_consensus.cc:3037] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:46.107407 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.096999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.222.3:52797 (local address 127.0.222.1:36677)
0924 00:54:46.097189 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.097219 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.101738 (+ 4519us) server_negotiation.cc:407] Connection header received
0924 00:54:46.101891 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.101897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.101985 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.102089 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:54:46.102701 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.103654 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.105713 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.106559 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.106673 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.106777 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.107051 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.107178 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:54:46.107481 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.096419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.222.3:52797)
0924 00:54:46.101637 (+ 5218us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.101652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:54:46.101774 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:54:46.102120 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:54:46.102124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:54:46.102146 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:54:46.102519 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.102532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.103786 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.103791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.105470 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:54:46.105487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.106719 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.106725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:54:46.106900 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.106927 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 00:54:46.107009 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:54:46.107150 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5106,"thread_start_us":4907,"threads_started":1}
I20260924 00:54:46.108284 21459 raft_consensus.cc:504] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.108323 21129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbda32dbdf24df6b6c45131b6e51870" candidate_uuid: "961b6d294f9f4330aed4072854228c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fb6f393d6541d3935585c95ce2ffb1" is_pre_election: true
I20260924 00:54:46.108806 21129 raft_consensus.cc:2454] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 961b6d294f9f4330aed4072854228c9f in term 0.
I20260924 00:54:46.108891 21459 leader_election.cc:288] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [CANDIDATE]: Term 1 election: Requested vote from peers bac709ce86144eb0b6fd01059c876503 (127.0.222.2:40047), 76fb6f393d6541d3935585c95ce2ffb1 (127.0.222.1:36677)
I20260924 00:54:46.109337 21260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbda32dbdf24df6b6c45131b6e51870" candidate_uuid: "961b6d294f9f4330aed4072854228c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac709ce86144eb0b6fd01059c876503"
I20260924 00:54:46.109401 21129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cbda32dbdf24df6b6c45131b6e51870" candidate_uuid: "961b6d294f9f4330aed4072854228c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fb6f393d6541d3935585c95ce2ffb1"
I20260924 00:54:46.109570 21129 raft_consensus.cc:3037] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:46.109562 21260 raft_consensus.cc:3037] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:46.112713 21129 raft_consensus.cc:2454] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 961b6d294f9f4330aed4072854228c9f in term 1.
I20260924 00:54:46.112723 21260 raft_consensus.cc:2454] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 961b6d294f9f4330aed4072854228c9f in term 1.
I20260924 00:54:46.113126 21326 leader_election.cc:302] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76fb6f393d6541d3935585c95ce2ffb1, 961b6d294f9f4330aed4072854228c9f; no voters:
I20260924 00:54:46.113477 21459 raft_consensus.cc:2781] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:46.113976 21459 raft_consensus.cc:686] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 1 LEADER]: Becoming Leader. State: Replica: 961b6d294f9f4330aed4072854228c9f, State: Running, Role: LEADER
I20260924 00:54:46.114463 21459 consensus_queue.cc:237] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } }
I20260924 00:54:46.118713 20988 catalog_manager.cc:5495] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 961b6d294f9f4330aed4072854228c9f (127.0.222.3). New cstate: current_term: 1 leader_uuid: "961b6d294f9f4330aed4072854228c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "961b6d294f9f4330aed4072854228c9f" member_type: VOTER last_known_addr { host: "127.0.222.3" port: 41037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 } health_report { overall_health: UNKNOWN } } }
I20260924 00:54:46.136188 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.130713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.0.222.3:41037)
0924 00:54:46.130930 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.130985 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.131030 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:54:46.131270 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.131274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.131366 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.131497 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:54:46.131903 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.132649 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.134065 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.134481 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.134591 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.135077 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:54:46.135614 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:54:46.135745 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.135859 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.135950 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:54:46.144637 21260 raft_consensus.cc:1261] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 1 FOLLOWER]: Refusing update from remote peer 961b6d294f9f4330aed4072854228c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:46.144634 21129 raft_consensus.cc:1261] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 1 FOLLOWER]: Refusing update from remote peer 961b6d294f9f4330aed4072854228c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:54:46.145785 21459 consensus_queue.cc:1035] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fb6f393d6541d3935585c95ce2ffb1" member_type: VOTER last_known_addr { host: "127.0.222.1" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:46.146122 21463 consensus_queue.cc:1035] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac709ce86144eb0b6fd01059c876503" member_type: VOTER last_known_addr { host: "127.0.222.2" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:54:46.159618 21467 mvcc.cc:205] Tried to move back new op lower bound from 7332705428037472256 to 7332705427928576000. Current Snapshot: MvccSnapshot[applied={T|T < 7332705428037472256}]
I20260924 00:54:46.171929 20988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:46.172191 20988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:54:46.175535 20988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46019f6701ef4934b95a773141330b1e: Sending DeleteTablet for 3 replicas of tablet 3cbda32dbdf24df6b6c45131b6e51870
I20260924 00:54:46.176611 21240 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbda32dbdf24df6b6c45131b6e51870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:46 UTC) from {username='slave'} at 127.0.0.1:42078
I20260924 00:54:46.176679 21370 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbda32dbdf24df6b6c45131b6e51870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:46 UTC) from {username='slave'} at 127.0.0.1:43858
I20260924 00:54:46.176673 21109 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cbda32dbdf24df6b6c45131b6e51870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:54:46 UTC) from {username='slave'} at 127.0.0.1:59324
I20260924 00:54:46.178645 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21047
I20260924 00:54:46.189523 21476 tablet_replica.cc:330] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: stopping tablet replica
I20260924 00:54:46.190069 21476 raft_consensus.cc:2229] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:46.190647 21476 raft_consensus.cc:2258] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:46.193405 21476 ts_tablet_manager.cc:1905] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:46.207896 21478 tablet_replica.cc:330] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1: stopping tablet replica
I20260924 00:54:46.208412 21478 raft_consensus.cc:2229] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:54:46.209004 21478 raft_consensus.cc:2258] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:46.210602 21476 ts_tablet_manager.cc:1918] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:46.211351 21477 tablet_replica.cc:330] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: stopping tablet replica
I20260924 00:54:46.211838 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.180105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.0.222.1:36677)
0924 00:54:46.180291 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.180297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.191849 (+ 11552us) server_negotiation.cc:407] Connection header received
0924 00:54:46.197691 (+ 5842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.197698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.197820 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.197904 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:46.198689 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.199924 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.205573 (+ 5649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.206277 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.206386 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.210128 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:46.210139 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:46.210157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:46.210929 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:46.210996 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:46.211024 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:46.211139 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.211358 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.211501 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 00:54:46.212248 21476 log.cc:1196] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-1/wal/wals/3cbda32dbdf24df6b6c45131b6e51870
I20260924 00:54:46.212601 21478 ts_tablet_manager.cc:1905] T 3cbda32dbdf24df6b6c45131b6e51870 P 76fb6f393d6541d3935585c95ce2ffb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:46.213004 21169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:46.213521 21476 ts_tablet_manager.cc:1939] T 3cbda32dbdf24df6b6c45131b6e51870 P bac709ce86144eb0b6fd01059c876503: Deleting consensus metadata
I20260924 00:54:46.211864 21477 raft_consensus.cc:2229] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:54:46.214390 21477 raft_consensus.cc:2258] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:54:46.216084 20974 catalog_manager.cc:4841] TS bac709ce86144eb0b6fd01059c876503 (127.0.222.2:40047): tablet 3cbda32dbdf24df6b6c45131b6e51870 (table test-table [id=bdd9097d74dc4052b799aab422c1a3f4]) successfully deleted
I20260924 00:54:46.217109 21477 ts_tablet_manager.cc:1905] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:54:46.224876 21477 ts_tablet_manager.cc:1918] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:54:46.225075 21477 log.cc:1196] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: Deleting WAL directory at /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790210962623352-888-0/minicluster-data/ts-2/wal/wals/3cbda32dbdf24df6b6c45131b6e51870
I20260924 00:54:46.225641 21477 ts_tablet_manager.cc:1939] T 3cbda32dbdf24df6b6c45131b6e51870 P 961b6d294f9f4330aed4072854228c9f: Deleting consensus metadata
I20260924 00:54:46.227686 20975 catalog_manager.cc:4841] TS 961b6d294f9f4330aed4072854228c9f (127.0.222.3:41037): tablet 3cbda32dbdf24df6b6c45131b6e51870 (table test-table [id=bdd9097d74dc4052b799aab422c1a3f4]) successfully deleted
I20260924 00:54:46.304065 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 21047
I20260924 00:54:46.318228 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21178
W20260924 00:54:46.318393 20975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.222.1:36677 (error 108)
I20260924 00:54:46.320890 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.320388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.0.1:59344)
0924 00:54:46.320584 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.320709 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 00:54:46.321173 20975 catalog_manager.cc:4584] TS 76fb6f393d6541d3935585c95ce2ffb1 (127.0.222.1:36677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cbda32dbdf24df6b6c45131b6e51870: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36677: connect: Connection refused (error 111)
I20260924 00:54:46.326325 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.318988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.0.222.2:40047)
0924 00:54:46.319180 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.319186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.319262 (+ 76us) server_negotiation.cc:407] Connection header received
0924 00:54:46.319466 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.319472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.319594 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.319690 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:46.320322 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.321405 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.323244 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.323673 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.323764 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.324908 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:46.324916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:46.324930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:46.325650 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:46.325698 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:46.325723 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:46.325805 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.325951 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.326038 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 00:54:46.326990 21300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:46.375095 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.374525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.0.1:59346)
0924 00:54:46.374772 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.374896 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 00:54:46.376291 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.375872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.0.1:59348)
0924 00:54:46.376035 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.376134 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 00:54:46.416563 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 21178
I20260924 00:54:46.431852 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21308
I20260924 00:54:46.437038 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.436538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.0.1:59354)
0924 00:54:46.436749 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.436833 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 00:54:46.437986 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.437649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.0.1:59358)
0924 00:54:46.437789 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.437861 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 00:54:46.443576 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.432622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.0.222.3:41037)
0924 00:54:46.432823 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:46.432830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:46.432851 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:54:46.433021 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:46.433027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:46.433123 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:46.433224 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:46.433689 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.434475 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.435703 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:46.436143 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:46.436249 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:46.442113 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:46.442126 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:46.442141 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:46.442775 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:46.442827 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:46.442859 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:46.443040 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:46.443124 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 00:54:46.443242 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 00:54:46.444273 21430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:46.518554 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.518003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.0.1:59360)
0924 00:54:46.518247 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.518342 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 00:54:46.519697 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:46.519276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.1:36677 (local address 127.0.0.1:59374)
0924 00:54:46.519488 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 00:54:46.519584 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.1:36677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 00:54:46.563591 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 21308
I20260924 00:54:46.579756 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 20957
I20260924 00:54:46.580971 21018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:46.688584 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 20957
2026-09-24T00:54:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2749 ms)
[----------] 6 tests from EncryptionPolicyTest (21159 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:46 dist-test-slave-rtvz krb5kdc[21487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:46 dist-test-slave-rtvz krb5kdc[21487](info): set up 2 sockets
Sep 24 00:54:46 dist-test-slave-rtvz krb5kdc[21487](info): commencing operation
krb5kdc: starting...
W20260924 00:54:47.720679 888 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:47 dist-test-slave-rtvz krb5kdc[21487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:54:47.973564 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:48.299620 21500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:48.299908 21500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:48.300040 21500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:48.307292 21500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:48.307411 21500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:48.307466 21500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:48.307546 21500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:48.307596 21500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:48.317756 21500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:48.319067 21500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:48.320466 21500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:48.332089 21505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:48.332041 21506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:48.332126 21508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:48.333022 21500 server_base.cc:1034] running on GCE node
I20260924 00:54:48.333878 21500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:48.335283 21500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:48.336479 21500 hybrid_clock.cc:648] HybridClock initialized: now 1790211288336435 us; error 41 us; skew 500 ppm
I20260924 00:54:48.336800 21500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:48 dist-test-slave-rtvz krb5kdc[21487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:48.339838 21500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:54:48.341378 21500 webserver.cc:460] Webserver started at http://127.0.0.1:40075/ using document root <none> and password file <none>
I20260924 00:54:48.342103 21500 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:48.342222 21500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:48.342514 21500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:48.345472 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "0f6353004fa44ba9b1867b51dff8942b"
format_stamp: "Formatted at 2026-09-24 00:54:48 on dist-test-slave-rtvz"
server_key: "8fe1ae0305c11ddcd43e15c99fe26dee"
server_key_iv: "dad87fc4129c6cb77ac1e9cdc5de96fd"
server_key_version: "encryptionkey@0"
I20260924 00:54:48.346226 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "0f6353004fa44ba9b1867b51dff8942b"
format_stamp: "Formatted at 2026-09-24 00:54:48 on dist-test-slave-rtvz"
server_key: "8fe1ae0305c11ddcd43e15c99fe26dee"
server_key_iv: "dad87fc4129c6cb77ac1e9cdc5de96fd"
server_key_version: "encryptionkey@0"
I20260924 00:54:48.351475 21500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:54:48.354974 21515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:48.356276 21500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:54:48.356436 21500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "0f6353004fa44ba9b1867b51dff8942b"
format_stamp: "Formatted at 2026-09-24 00:54:48 on dist-test-slave-rtvz"
server_key: "8fe1ae0305c11ddcd43e15c99fe26dee"
server_key_iv: "dad87fc4129c6cb77ac1e9cdc5de96fd"
server_key_version: "encryptionkey@0"
I20260924 00:54:48.356607 21500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:48.370452 21500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:48.375041 21500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:48.375308 21500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:48.392839 21500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38619
I20260924 00:54:48.392870 21566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38619 every 8 connection(s)
I20260924 00:54:48.394258 21500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:48.397408 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 21500
I20260924 00:54:48.397627 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:48.398137 21567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:48.398267 888 external_mini_cluster.cc:1428] Setting key a5cb84292feb37f6fe143fe3b5c847c4
I20260924 00:54:48.411319 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b: Bootstrap starting.
Sep 24 00:54:48 dist-test-slave-rtvz krb5kdc[21487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:54:48.414681 21567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:48.415822 21567 log.cc:824] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:48.418769 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b: No bootstrap required, opened a new log
I20260924 00:54:48.421088 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:48.400196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39665 (local address 127.0.0.1:38619)
0924 00:54:48.401021 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:48.401041 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:48.401092 (+ 51us) server_negotiation.cc:407] Connection header received
0924 00:54:48.403108 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:48.403140 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:48.403710 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:48.404174 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:48.405622 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:48.406894 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:48.408173 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:48.408913 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:48.409096 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:48.413153 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:48.413161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:48.413176 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:48.413244 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:48.416318 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:48.417221 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:48.417231 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:48.417241 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:48.417413 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:48.417904 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:48.417907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:48.417909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:48.418442 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:48.418749 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:48.419262 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 00:54:48.419659 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":336,"threads_started":1}
I20260924 00:54:48.425657 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:54:48.428277 21567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6353004fa44ba9b1867b51dff8942b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } }
I20260924 00:54:48.428782 21567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:48.428887 21567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6353004fa44ba9b1867b51dff8942b, State: Initialized, Role: FOLLOWER
I20260924 00:54:48.429606 21567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6353004fa44ba9b1867b51dff8942b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } }
I20260924 00:54:48.429908 21567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:48.430109 21567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:48.430286 21567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:48.433022 21567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6353004fa44ba9b1867b51dff8942b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } }
I20260924 00:54:48.433525 21567 leader_election.cc:302] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f6353004fa44ba9b1867b51dff8942b; no voters:
I20260924 00:54:48.434175 21567 leader_election.cc:288] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:48.434273 21577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:48.436368 21567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:48.437466 21577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [term 1 LEADER]: Becoming Leader. State: Replica: 0f6353004fa44ba9b1867b51dff8942b, State: Running, Role: LEADER
I20260924 00:54:48.438016 21577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6353004fa44ba9b1867b51dff8942b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } }
I20260924 00:54:48.440519 21579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f6353004fa44ba9b1867b51dff8942b. Latest consensus state: current_term: 1 leader_uuid: "0f6353004fa44ba9b1867b51dff8942b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6353004fa44ba9b1867b51dff8942b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } } }
I20260924 00:54:48.440919 21579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:48.441735 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f6353004fa44ba9b1867b51dff8942b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6353004fa44ba9b1867b51dff8942b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } } }
I20260924 00:54:48.442603 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:48.447521 21584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:48.448705 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:48.431470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48595 (local address 127.0.0.1:38619)
0924 00:54:48.431666 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:48.431672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:48.431973 (+ 301us) server_negotiation.cc:407] Connection header received
0924 00:54:48.432123 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:48.432128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:48.432213 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:48.432323 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:48.433700 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:48.434855 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:48.436623 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:48.437270 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:48.438099 (+ 829us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:48.440665 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:48.440675 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:48.440681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:48.440726 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:48.445262 (+ 4536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:48.446257 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:48.446263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:48.446267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:48.446422 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:48.446986 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:48.446992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:48.446996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:48.447457 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:48.447773 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:48.448102 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 00:54:48.448304 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 00:54:48.452977 21584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:48.463285 21584 catalog_manager.cc:1347] Generated new cluster ID: 5e71a7fdb2fc42dc9db6ad2c20d08498
I20260924 00:54:48.463408 21584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:48.482780 21584 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:48.484094 21584 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:48.492499 21584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f6353004fa44ba9b1867b51dff8942b: Generated new TSK 0
I20260924 00:54:48.493410 21584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:48.506709 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21500
I20260924 00:54:48.507943 21562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:48.601259 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 21500
W20260924 00:54:48.615240 21485 connection.cc:535] client connection to 127.0.0.1:38619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38619): Connection reset by peer (error 104)
2026-09-24T00:54:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:48 dist-test-slave-rtvz krb5kdc[21599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:48 dist-test-slave-rtvz krb5kdc[21599](info): set up 2 sockets
Sep 24 00:54:48 dist-test-slave-rtvz krb5kdc[21599](info): commencing operation
krb5kdc: starting...
W20260924 00:54:49.624569 888 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:49 dist-test-slave-rtvz krb5kdc[21599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:54:49.877172 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:50.212922 21612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:50.213284 21612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:50.213420 21612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:50.220705 21612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:50.220834 21612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:50.220918 21612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:50.221006 21612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:50.221065 21612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:50.231096 21612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:50.232403 21612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:50.233769 21612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:50.245350 21620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:50.245369 21618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:50.245639 21617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:50.246351 21612 server_base.cc:1034] running on GCE node
I20260924 00:54:50.247190 21612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:50.248624 21612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:50.249840 21612 hybrid_clock.cc:648] HybridClock initialized: now 1790211290249786 us; error 38 us; skew 500 ppm
I20260924 00:54:50.250263 21612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:50 dist-test-slave-rtvz krb5kdc[21599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:50.253396 21612 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:54:50.254827 21612 webserver.cc:460] Webserver started at http://127.0.0.1:41575/ using document root <none> and password file <none>
I20260924 00:54:50.255487 21612 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:50.255586 21612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:50.255841 21612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:50.258882 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "7cb98ea2e0a7462aac343db963dcf233"
format_stamp: "Formatted at 2026-09-24 00:54:50 on dist-test-slave-rtvz"
server_key: "8e5c13394e1605541ad0ff407394d49f"
server_key_iv: "9fe40fc10b139bbc42dc1309e9d6de01"
server_key_version: "encryptionkey@0"
I20260924 00:54:50.259610 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "7cb98ea2e0a7462aac343db963dcf233"
format_stamp: "Formatted at 2026-09-24 00:54:50 on dist-test-slave-rtvz"
server_key: "8e5c13394e1605541ad0ff407394d49f"
server_key_iv: "9fe40fc10b139bbc42dc1309e9d6de01"
server_key_version: "encryptionkey@0"
I20260924 00:54:50.265105 21612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 00:54:50.268747 21627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:50.270295 21612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 00:54:50.270459 21612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "7cb98ea2e0a7462aac343db963dcf233"
format_stamp: "Formatted at 2026-09-24 00:54:50 on dist-test-slave-rtvz"
server_key: "8e5c13394e1605541ad0ff407394d49f"
server_key_iv: "9fe40fc10b139bbc42dc1309e9d6de01"
server_key_version: "encryptionkey@0"
I20260924 00:54:50.270658 21612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:50.293977 21612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:50.298660 21612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:50.298938 21612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:50.316936 21612 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43049
I20260924 00:54:50.317028 21679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43049 every 8 connection(s)
I20260924 00:54:50.318306 21612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:50.321955 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:50.322356 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 21612
I20260924 00:54:50.322518 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:50.322911 888 external_mini_cluster.cc:1428] Setting key a4763913643c2f7e30fad56a59befeb5
I20260924 00:54:50.336092 21680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233: Bootstrap starting.
I20260924 00:54:50.338260 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:50.324701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57333 (local address 127.0.0.1:43049)
0924 00:54:50.325543 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:50.325559 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:50.325609 (+ 50us) server_negotiation.cc:407] Connection header received
0924 00:54:50.328166 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:50.328201 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:50.328751 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:50.329247 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:50.329762 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:50.331630 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:50.333069 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:50.334212 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:50.334378 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:50.335521 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:50.335531 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:50.335550 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:50.336247 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:50.336346 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:50.336384 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:50.336540 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:50.336762 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 00:54:50.336935 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":398,"threads_started":1}
I20260924 00:54:50.340018 21680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:50.341323 21680 log.cc:824] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:50.342620 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:54:50.344334 21680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233: No bootstrap required, opened a new log
I20260924 00:54:50.353432 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:50.346883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37677 (local address 127.0.0.1:43049)
0924 00:54:50.347069 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:50.347075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:50.347373 (+ 298us) server_negotiation.cc:407] Connection header received
0924 00:54:50.347542 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:50.347547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:50.347619 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:50.347735 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:50.348260 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:50.349454 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:50.350833 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:50.351529 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:50.351650 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:50.352700 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:50.352707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:50.352711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:50.352732 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:50.352775 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:50.352795 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:50.352915 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:50.353056 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 00:54:50.353116 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 00:54:50.353093 21680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb98ea2e0a7462aac343db963dcf233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } }
I20260924 00:54:50.353663 21680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:50.353765 21680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb98ea2e0a7462aac343db963dcf233, State: Initialized, Role: FOLLOWER
I20260924 00:54:50.354466 21680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb98ea2e0a7462aac343db963dcf233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } }
I20260924 00:54:50.354768 21680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:50.354914 21680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:50.355074 21680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:50.357323 21680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb98ea2e0a7462aac343db963dcf233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } }
I20260924 00:54:50.357837 21680 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cb98ea2e0a7462aac343db963dcf233; no voters:
I20260924 00:54:50.358449 21680 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:50.358567 21690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:50.359251 21690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [term 1 LEADER]: Becoming Leader. State: Replica: 7cb98ea2e0a7462aac343db963dcf233, State: Running, Role: LEADER
I20260924 00:54:50.359951 21690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb98ea2e0a7462aac343db963dcf233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } }
I20260924 00:54:50.360487 21680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:50.363514 21692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb98ea2e0a7462aac343db963dcf233. Latest consensus state: current_term: 1 leader_uuid: "7cb98ea2e0a7462aac343db963dcf233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb98ea2e0a7462aac343db963dcf233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } } }
I20260924 00:54:50.363945 21692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:50.364454 21691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cb98ea2e0a7462aac343db963dcf233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb98ea2e0a7462aac343db963dcf233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } } }
I20260924 00:54:50.364742 21691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:50.365525 21699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:50.370985 21699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:50.380146 21699 catalog_manager.cc:1347] Generated new cluster ID: 60dc35378bc742c3b9dae40e0df69694
I20260924 00:54:50.380271 21699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:50.399144 21699 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:50.401118 21699 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:50.411181 21699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cb98ea2e0a7462aac343db963dcf233: Generated new TSK 0
I20260924 00:54:50.412266 21699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:50.424150 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21612
W20260924 00:54:50.425602 21675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57333
W20260924 00:54:50.426486 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21612: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:54:50.426680 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 21612
W20260924 00:54:50.440605 21597 connection.cc:535] client connection to 127.0.0.1:43049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43049): Connection reset by peer (error 104)
2026-09-24T00:54:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1825 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:50 dist-test-slave-rtvz krb5kdc[21712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:50 dist-test-slave-rtvz krb5kdc[21712](info): set up 2 sockets
Sep 24 00:54:50 dist-test-slave-rtvz krb5kdc[21712](info): commencing operation
krb5kdc: starting...
W20260924 00:54:51.460613 888 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:51 dist-test-slave-rtvz krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:54:51.713392 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:52.050525 21725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:52.050843 21725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:52.050964 21725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:52.058385 21725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:52.058506 21725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:52.058562 21725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:52.058648 21725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:52.058702 21725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:52.068843 21725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:52.070253 21725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:52.071599 21725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:52.083815 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:52.083798 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:52.083845 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:52.084199 21725 server_base.cc:1034] running on GCE node
I20260924 00:54:52.085506 21725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:52.087126 21725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:52.088349 21725 hybrid_clock.cc:648] HybridClock initialized: now 1790211292088295 us; error 46 us; skew 500 ppm
I20260924 00:54:52.088713 21725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:52 dist-test-slave-rtvz krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:52.091900 21725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:54:52.093452 21725 webserver.cc:460] Webserver started at http://127.0.0.1:38301/ using document root <none> and password file <none>
I20260924 00:54:52.094164 21725 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:52.094285 21725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:52.094581 21725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:52.097800 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "707b08c0441a4220b009760e0579db91"
format_stamp: "Formatted at 2026-09-24 00:54:52 on dist-test-slave-rtvz"
server_key: "dd43dbdfbedd1256486e29b6dd456c14"
server_key_iv: "4c29a2327e8531ca7e123e42cae83a39"
server_key_version: "encryptionkey@0"
I20260924 00:54:52.098572 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "707b08c0441a4220b009760e0579db91"
format_stamp: "Formatted at 2026-09-24 00:54:52 on dist-test-slave-rtvz"
server_key: "dd43dbdfbedd1256486e29b6dd456c14"
server_key_iv: "4c29a2327e8531ca7e123e42cae83a39"
server_key_version: "encryptionkey@0"
I20260924 00:54:52.104027 21725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:54:52.107686 21740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:52.109138 21725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 00:54:52.109338 21725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "707b08c0441a4220b009760e0579db91"
format_stamp: "Formatted at 2026-09-24 00:54:52 on dist-test-slave-rtvz"
server_key: "dd43dbdfbedd1256486e29b6dd456c14"
server_key_iv: "4c29a2327e8531ca7e123e42cae83a39"
server_key_version: "encryptionkey@0"
I20260924 00:54:52.109529 21725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:52.115514 21725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:52.119616 21725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:52.119902 21725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:52.137055 21725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40665
I20260924 00:54:52.137082 21792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40665 every 8 connection(s)
I20260924 00:54:52.138581 21725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:52.142326 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:52.148173 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 21725
I20260924 00:54:52.148350 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:52.148788 888 external_mini_cluster.cc:1428] Setting key f769f1f594f7387c6244039cf76f463e
I20260924 00:54:52.153473 21793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91: Bootstrap starting.
I20260924 00:54:52.156577 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:52.150575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55957 (local address 127.0.0.1:40665)
0924 00:54:52.151442 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:52.151454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:52.151500 (+ 46us) server_negotiation.cc:407] Connection header received
0924 00:54:52.153190 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:52.153265 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:52.153650 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:52.154006 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:52.155037 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:52.155044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:52.155062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:52.155077 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:52.155141 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:52.155164 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:52.155347 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:52.155453 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 00:54:52.155655 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":383,"threads_started":1}
I20260924 00:54:52.157140 21793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:52.158361 21793 log.cc:824] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:52.160537 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:54:52.161448 21793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91: No bootstrap required, opened a new log
I20260924 00:54:52.167771 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:52.165421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38149 (local address 127.0.0.1:40665)
0924 00:54:52.165609 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:52.165619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:52.166068 (+ 449us) server_negotiation.cc:407] Connection header received
0924 00:54:52.166215 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:52.166220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:52.166283 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:52.166385 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:52.167167 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:52.167172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:52.167176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:52.167193 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:52.167242 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:52.167263 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:52.167385 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:52.167429 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 00:54:52.167486 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 00:54:52.170339 21793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707b08c0441a4220b009760e0579db91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40665 } }
I20260924 00:54:52.170704 21793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:52.170775 21793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707b08c0441a4220b009760e0579db91, State: Initialized, Role: FOLLOWER
I20260924 00:54:52.171303 21793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707b08c0441a4220b009760e0579db91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40665 } }
I20260924 00:54:52.171522 21793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:52.171654 21793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:52.171839 21793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:52.174067 21793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707b08c0441a4220b009760e0579db91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40665 } }
I20260924 00:54:52.174551 21793 leader_election.cc:302] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 707b08c0441a4220b009760e0579db91; no voters:
I20260924 00:54:52.175233 21793 leader_election.cc:288] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:52.175359 21803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:52.175940 21803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [term 1 LEADER]: Becoming Leader. State: Replica: 707b08c0441a4220b009760e0579db91, State: Running, Role: LEADER
I20260924 00:54:52.176465 21803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707b08c0441a4220b009760e0579db91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40665 } }
I20260924 00:54:52.177186 21793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:52.179353 21804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "707b08c0441a4220b009760e0579db91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707b08c0441a4220b009760e0579db91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40665 } } }
I20260924 00:54:52.179759 21804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:52.179309 21805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707b08c0441a4220b009760e0579db91. Latest consensus state: current_term: 1 leader_uuid: "707b08c0441a4220b009760e0579db91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707b08c0441a4220b009760e0579db91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40665 } } }
I20260924 00:54:52.180321 21805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91 [sys.catalog]: This master's current role is: LEADER
W20260924 00:54:52.186687 21819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:54:52.186802 21819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:54:52.186892 21812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:52.188470 21812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:52.196393 21812 catalog_manager.cc:1347] Generated new cluster ID: 4ba4eb90c6804e0cbee1b5691b0d4853
I20260924 00:54:52.196520 21812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:52.211808 21812 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:52.212318 21812 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:52.222143 21812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 707b08c0441a4220b009760e0579db91: Generated new TSK 0
I20260924 00:54:52.223001 21812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:52.236742 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21725
W20260924 00:54:52.239106 21788 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55957
W20260924 00:54:52.240271 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:54:52.240412 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 21725
2026-09-24T00:54:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1815 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:52 dist-test-slave-rtvz krb5kdc[21824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:52 dist-test-slave-rtvz krb5kdc[21824](info): set up 2 sockets
Sep 24 00:54:52 dist-test-slave-rtvz krb5kdc[21824](info): commencing operation
krb5kdc: starting...
W20260924 00:54:53.261471 888 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:53 dist-test-slave-rtvz krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:54:53.516789 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:53.840296 21837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:53.840596 21837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:53.840716 21837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:53.848387 21837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:53.848505 21837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:53.848580 21837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:53.848662 21837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:53.848708 21837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:53.858773 21837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:53.860109 21837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:53.861730 21837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:53.874301 21843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:53.874338 21845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:53.874251 21842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:53.875406 21837 server_base.cc:1034] running on GCE node
I20260924 00:54:53.876336 21837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:53.877931 21837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:53.879159 21837 hybrid_clock.cc:648] HybridClock initialized: now 1790211293879125 us; error 37 us; skew 500 ppm
I20260924 00:54:53.879556 21837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:53 dist-test-slave-rtvz krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:53.882900 21837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:54:53.884743 21837 webserver.cc:460] Webserver started at http://127.0.0.1:36595/ using document root <none> and password file <none>
I20260924 00:54:53.885538 21837 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:53.885666 21837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:53.885970 21837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:53.889073 21837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "34e20c1983634da3b96b67e5df9fecaf"
format_stamp: "Formatted at 2026-09-24 00:54:53 on dist-test-slave-rtvz"
server_key: "454eff513d06f4977d2fb2e3709b476b"
server_key_iv: "b3953659cc6c5221d614464897d3ebe9"
server_key_version: "encryptionkey@0"
I20260924 00:54:53.889887 21837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "34e20c1983634da3b96b67e5df9fecaf"
format_stamp: "Formatted at 2026-09-24 00:54:53 on dist-test-slave-rtvz"
server_key: "454eff513d06f4977d2fb2e3709b476b"
server_key_iv: "b3953659cc6c5221d614464897d3ebe9"
server_key_version: "encryptionkey@0"
I20260924 00:54:53.895071 21837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:54:53.898587 21852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:53.900054 21837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:54:53.900245 21837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "34e20c1983634da3b96b67e5df9fecaf"
format_stamp: "Formatted at 2026-09-24 00:54:53 on dist-test-slave-rtvz"
server_key: "454eff513d06f4977d2fb2e3709b476b"
server_key_iv: "b3953659cc6c5221d614464897d3ebe9"
server_key_version: "encryptionkey@0"
I20260924 00:54:53.900454 21837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:53.919153 21837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:53.923674 21837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:53.924070 21837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:53.942348 21837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44437
I20260924 00:54:53.942371 21903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44437 every 8 connection(s)
I20260924 00:54:53.943751 21837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:53.947522 21904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:53.950994 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 21837
I20260924 00:54:53.951172 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:53.951571 888 external_mini_cluster.cc:1428] Setting key 6f64d57b172cdebd570598c95ab16d41
I20260924 00:54:53.960366 21904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf: Bootstrap starting.
I20260924 00:54:53.963938 21904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:53.964933 21904 log.cc:824] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf: Log is configured to *not* fsync() on all Append() calls
Sep 24 00:54:53 dist-test-slave-rtvz krb5kdc[21824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:54:53.968099 21904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf: No bootstrap required, opened a new log
I20260924 00:54:53.976938 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:53.953463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37227 (local address 127.0.0.1:44437)
0924 00:54:53.954258 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:53.954273 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:53.954324 (+ 51us) server_negotiation.cc:407] Connection header received
0924 00:54:53.956210 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:53.956242 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:53.956820 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:53.957315 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:53.958643 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:53.960554 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:53.961965 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:53.962991 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:53.963183 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:53.967696 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:53.967708 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:53.967730 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:53.967780 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:53.971749 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:53.972654 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:53.972664 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:53.972673 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:53.972909 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:53.973401 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:53.973406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:53.973410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:53.974252 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:53.974612 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:53.974959 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 00:54:53.975450 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":336,"threads_started":1}
I20260924 00:54:53.978765 21904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e20c1983634da3b96b67e5df9fecaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44437 } }
I20260924 00:54:53.979269 21904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:53.979380 21904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e20c1983634da3b96b67e5df9fecaf, State: Initialized, Role: FOLLOWER
I20260924 00:54:53.980012 21904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e20c1983634da3b96b67e5df9fecaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44437 } }
I20260924 00:54:53.980262 21904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:53.980410 21904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:53.980604 21904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:53.981361 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:54:53.982977 21904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e20c1983634da3b96b67e5df9fecaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44437 } }
I20260924 00:54:53.983469 21904 leader_election.cc:302] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34e20c1983634da3b96b67e5df9fecaf; no voters:
I20260924 00:54:53.984088 21904 leader_election.cc:288] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:53.984184 21910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:53.991111 21910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [term 1 LEADER]: Becoming Leader. State: Replica: 34e20c1983634da3b96b67e5df9fecaf, State: Running, Role: LEADER
I20260924 00:54:53.991822 21910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e20c1983634da3b96b67e5df9fecaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44437 } }
I20260924 00:54:53.993124 21904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:53.995230 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34e20c1983634da3b96b67e5df9fecaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e20c1983634da3b96b67e5df9fecaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44437 } } }
I20260924 00:54:53.996774 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:53.995891 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34e20c1983634da3b96b67e5df9fecaf. Latest consensus state: current_term: 1 leader_uuid: "34e20c1983634da3b96b67e5df9fecaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34e20c1983634da3b96b67e5df9fecaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44437 } } }
I20260924 00:54:53.997629 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:53.997925 21920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:54.000921 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:53.985711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52467 (local address 127.0.0.1:44437)
0924 00:54:53.985894 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:53.985901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:53.986298 (+ 397us) server_negotiation.cc:407] Connection header received
0924 00:54:53.986458 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:53.986463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:53.986534 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:53.986615 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:53.987996 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:53.988735 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:53.990022 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:53.990452 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:53.990565 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:53.993080 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:53.993085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:53.993088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:53.993125 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:53.996696 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:53.997715 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:53.997719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:53.997723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:53.997863 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:53.998357 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:53.998362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:53.998365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:53.998806 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:54.000203 (+ 1397us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:54.000344 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 00:54:54.000572 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:54:54.004282 21920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:54.013221 21920 catalog_manager.cc:1347] Generated new cluster ID: e032b3cb95df4850804e2c593fa3815a
I20260924 00:54:54.013368 21920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:54.022941 21920 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:54.024847 21920 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:54.041874 21920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34e20c1983634da3b96b67e5df9fecaf: Generated new TSK 0
I20260924 00:54:54.042907 21920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:54.057718 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21837
I20260924 00:54:54.059216 21899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:54.151120 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 21837
2026-09-24T00:54:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:54 dist-test-slave-rtvz krb5kdc[21937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:54 dist-test-slave-rtvz krb5kdc[21937](info): set up 2 sockets
Sep 24 00:54:54 dist-test-slave-rtvz krb5kdc[21937](info): commencing operation
krb5kdc: starting...
W20260924 00:54:55.172624 888 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:55 dist-test-slave-rtvz krb5kdc[21937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:54:55.429541 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:55.762579 21950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:55.763037 21950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:55.763191 21950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:55.770964 21950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:55.771098 21950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:55.771179 21950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:55.771271 21950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:55.771337 21950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:55.782078 21950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:55.783458 21950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:55.784758 21950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:55.796188 21956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:55.796207 21958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:55.796686 21955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:55.797688 21950 server_base.cc:1034] running on GCE node
I20260924 00:54:55.798493 21950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:55.799948 21950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:55.801172 21950 hybrid_clock.cc:648] HybridClock initialized: now 1790211295801121 us; error 45 us; skew 500 ppm
I20260924 00:54:55.801568 21950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:55 dist-test-slave-rtvz krb5kdc[21937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:55.804723 21950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:54:55.806314 21950 webserver.cc:460] Webserver started at http://127.0.0.1:34375/ using document root <none> and password file <none>
I20260924 00:54:55.807015 21950 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:55.807119 21950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:55.807387 21950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:55.810580 21950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "d767923ee2504c508603c675eb96210e"
format_stamp: "Formatted at 2026-09-24 00:54:55 on dist-test-slave-rtvz"
server_key: "0f7169f487267fbd68883496fc6b2a0a"
server_key_iv: "386488e0bf53c34af799c76959029725"
server_key_version: "encryptionkey@0"
I20260924 00:54:55.811338 21950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "d767923ee2504c508603c675eb96210e"
format_stamp: "Formatted at 2026-09-24 00:54:55 on dist-test-slave-rtvz"
server_key: "0f7169f487267fbd68883496fc6b2a0a"
server_key_iv: "386488e0bf53c34af799c76959029725"
server_key_version: "encryptionkey@0"
I20260924 00:54:55.816689 21950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:54:55.820616 21965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:55.822117 21950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:55.822283 21950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "d767923ee2504c508603c675eb96210e"
format_stamp: "Formatted at 2026-09-24 00:54:55 on dist-test-slave-rtvz"
server_key: "0f7169f487267fbd68883496fc6b2a0a"
server_key_iv: "386488e0bf53c34af799c76959029725"
server_key_version: "encryptionkey@0"
I20260924 00:54:55.822481 21950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:55.834828 21950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:55.839108 21950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:55.839367 21950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:55.856886 21950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46469
I20260924 00:54:55.856920 22016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46469 every 8 connection(s)
I20260924 00:54:55.858381 21950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:55.862279 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:55.863708 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 21950
I20260924 00:54:55.863906 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:55.864445 888 external_mini_cluster.cc:1428] Setting key 255b43dead0c559742a21ebcd6410020
I20260924 00:54:55.876499 22017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e: Bootstrap starting.
I20260924 00:54:55.878165 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:55.866411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33597 (local address 127.0.0.1:46469)
0924 00:54:55.867219 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:55.867232 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:55.867275 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:54:55.868892 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:55.868919 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:55.869408 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:55.869754 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:55.870345 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:55.871692 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:55.873611 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:55.874363 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:55.874494 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:55.875724 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:55.875732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:55.875746 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:55.876334 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:55.876424 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:55.876450 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:55.876551 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:55.876784 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 00:54:55.877081 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":348,"threads_started":1}
I20260924 00:54:55.879876 22017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:55.880987 22017 log.cc:824] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:55.881989 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:54:55.884369 22017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e: No bootstrap required, opened a new log
I20260924 00:54:55.894112 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:55.887217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50877 (local address 127.0.0.1:46469)
0924 00:54:55.887408 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:55.887418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:55.887984 (+ 566us) server_negotiation.cc:407] Connection header received
0924 00:54:55.888154 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:55.888158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:55.888224 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:55.888308 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:55.888934 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:55.889727 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:55.891555 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:55.892026 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:55.892127 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:55.893283 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:55.893288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:55.893291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:55.893311 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:55.893352 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:55.893368 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:55.893448 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:55.893654 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 00:54:55.893740 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 00:54:55.894208 22017 raft_consensus.cc:348] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d767923ee2504c508603c675eb96210e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } }
I20260924 00:54:55.894711 22017 raft_consensus.cc:374] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:55.894811 22017 raft_consensus.cc:729] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d767923ee2504c508603c675eb96210e, State: Initialized, Role: FOLLOWER
I20260924 00:54:55.895414 22017 consensus_queue.cc:260] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d767923ee2504c508603c675eb96210e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } }
I20260924 00:54:55.895682 22017 raft_consensus.cc:388] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:55.895818 22017 raft_consensus.cc:482] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:55.895974 22017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:55.898289 22017 raft_consensus.cc:504] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d767923ee2504c508603c675eb96210e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } }
I20260924 00:54:55.898878 22017 leader_election.cc:302] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d767923ee2504c508603c675eb96210e; no voters:
I20260924 00:54:55.899506 22017 leader_election.cc:288] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:55.899641 22027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:55.900286 22027 raft_consensus.cc:686] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [term 1 LEADER]: Becoming Leader. State: Replica: d767923ee2504c508603c675eb96210e, State: Running, Role: LEADER
I20260924 00:54:55.900830 22027 consensus_queue.cc:237] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d767923ee2504c508603c675eb96210e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } }
I20260924 00:54:55.901614 22017 sys_catalog.cc:564] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:55.903622 22029 sys_catalog.cc:455] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d767923ee2504c508603c675eb96210e. Latest consensus state: current_term: 1 leader_uuid: "d767923ee2504c508603c675eb96210e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d767923ee2504c508603c675eb96210e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } } }
I20260924 00:54:55.903975 22029 sys_catalog.cc:458] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:55.903578 22028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d767923ee2504c508603c675eb96210e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d767923ee2504c508603c675eb96210e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } } }
I20260924 00:54:55.904908 22028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:55.906342 22034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:55.911448 22034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:55.920573 22034 catalog_manager.cc:1347] Generated new cluster ID: 9b3915f218f642b29445f22a522eaf9a
I20260924 00:54:55.920702 22034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:55.935207 22034 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:55.936592 22034 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:55.946256 22034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d767923ee2504c508603c675eb96210e: Generated new TSK 0
I20260924 00:54:55.947125 22034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:55.960246 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21950
W20260924 00:54:55.961555 22012 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33597
W20260924 00:54:55.962446 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 21950: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:54:55.962627 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 21950
2026-09-24T00:54:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1814 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:56 dist-test-slave-rtvz krb5kdc[22048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:56 dist-test-slave-rtvz krb5kdc[22048](info): set up 2 sockets
Sep 24 00:54:56 dist-test-slave-rtvz krb5kdc[22048](info): commencing operation
krb5kdc: starting...
W20260924 00:54:56.992657 888 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:57 dist-test-slave-rtvz krb5kdc[22048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:54:57.244361 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:57.566790 22061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:57.567096 22061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:57.567216 22061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:57.574609 22061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:57.574739 22061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:57.574793 22061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:57.574878 22061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:57.574934 22061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:57.585004 22061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:57.586362 22061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:57.587649 22061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:57.598949 22070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:57.599043 22067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:57.599067 22068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:57.600121 22061 server_base.cc:1034] running on GCE node
I20260924 00:54:57.600921 22061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:57.602366 22061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:57.603564 22061 hybrid_clock.cc:648] HybridClock initialized: now 1790211297603525 us; error 31 us; skew 500 ppm
I20260924 00:54:57.603891 22061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:57 dist-test-slave-rtvz krb5kdc[22048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:57.612315 22061 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:54:57.614135 22061 webserver.cc:460] Webserver started at http://127.0.0.1:42089/ using document root <none> and password file <none>
I20260924 00:54:57.614835 22061 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:57.614960 22061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:57.615276 22061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:57.618327 22061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "18236cb574cf441996705f67ef1b44cd"
format_stamp: "Formatted at 2026-09-24 00:54:57 on dist-test-slave-rtvz"
server_key: "28077cb496c34e613d0e91b22490c245"
server_key_iv: "f3970a59929420bfecc3b78dd7ef96e1"
server_key_version: "encryptionkey@0"
I20260924 00:54:57.619069 22061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "18236cb574cf441996705f67ef1b44cd"
format_stamp: "Formatted at 2026-09-24 00:54:57 on dist-test-slave-rtvz"
server_key: "28077cb496c34e613d0e91b22490c245"
server_key_iv: "f3970a59929420bfecc3b78dd7ef96e1"
server_key_version: "encryptionkey@0"
I20260924 00:54:57.624431 22061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:54:57.627885 22077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:57.629300 22061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:57.629464 22061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "18236cb574cf441996705f67ef1b44cd"
format_stamp: "Formatted at 2026-09-24 00:54:57 on dist-test-slave-rtvz"
server_key: "28077cb496c34e613d0e91b22490c245"
server_key_iv: "f3970a59929420bfecc3b78dd7ef96e1"
server_key_version: "encryptionkey@0"
I20260924 00:54:57.629647 22061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:57.635325 22061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:57.639254 22061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:57.639515 22061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:57.657073 22061 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41359
I20260924 00:54:57.657107 22128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41359 every 8 connection(s)
I20260924 00:54:57.658831 22061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:57.662845 22129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:57.668264 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22061
I20260924 00:54:57.668450 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:57.668963 888 external_mini_cluster.cc:1428] Setting key 022d569ebce9644b1724bb980ebae86f
I20260924 00:54:57.674587 22129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd: Bootstrap starting.
I20260924 00:54:57.677424 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:57.670902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56517 (local address 127.0.0.1:41359)
0924 00:54:57.671774 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:57.671787 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:57.671836 (+ 49us) server_negotiation.cc:407] Connection header received
0924 00:54:57.673757 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:57.673784 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:57.674170 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:57.674525 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:57.675607 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:57.675615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:57.675629 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:57.675646 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:57.675725 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:57.675749 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:57.675959 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:57.676068 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 00:54:57.676291 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":361,"threads_started":1}
I20260924 00:54:57.678128 22129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:57.679159 22129 log.cc:824] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:57.681372 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:54:57.682348 22129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd: No bootstrap required, opened a new log
I20260924 00:54:57.687999 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:57.685528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52573 (local address 127.0.0.1:41359)
0924 00:54:57.685715 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:57.685721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:57.686178 (+ 457us) server_negotiation.cc:407] Connection header received
0924 00:54:57.686312 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:57.686317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:57.686391 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:57.686494 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:57.687322 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:57.687328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:57.687333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:54:57.687352 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:57.687404 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:54:57.687424 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:57.687603 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:57.687646 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 00:54:57.687706 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:54:57.690593 22129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18236cb574cf441996705f67ef1b44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260924 00:54:57.690943 22129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:57.691025 22129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18236cb574cf441996705f67ef1b44cd, State: Initialized, Role: FOLLOWER
I20260924 00:54:57.691530 22129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18236cb574cf441996705f67ef1b44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260924 00:54:57.691735 22129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:57.691831 22129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:57.691957 22129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:57.694223 22129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18236cb574cf441996705f67ef1b44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260924 00:54:57.694716 22129 leader_election.cc:302] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18236cb574cf441996705f67ef1b44cd; no voters:
I20260924 00:54:57.695355 22129 leader_election.cc:288] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:57.695515 22139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:57.696244 22139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [term 1 LEADER]: Becoming Leader. State: Replica: 18236cb574cf441996705f67ef1b44cd, State: Running, Role: LEADER
I20260924 00:54:57.696862 22139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18236cb574cf441996705f67ef1b44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260924 00:54:57.697448 22129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:57.700315 22141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18236cb574cf441996705f67ef1b44cd. Latest consensus state: current_term: 1 leader_uuid: "18236cb574cf441996705f67ef1b44cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18236cb574cf441996705f67ef1b44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } } }
I20260924 00:54:57.700695 22141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:57.701277 22140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18236cb574cf441996705f67ef1b44cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18236cb574cf441996705f67ef1b44cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } } }
I20260924 00:54:57.701571 22140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:57.701715 22146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:57.709681 22146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:57.720256 22146 catalog_manager.cc:1347] Generated new cluster ID: 94220348e2414a909c99c1a9d502ca17
I20260924 00:54:57.720387 22146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:57.752967 22146 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:57.753517 22146 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:57.776793 22146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18236cb574cf441996705f67ef1b44cd: Generated new TSK 0
I20260924 00:54:57.777949 22146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:57.799451 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22061
W20260924 00:54:57.800906 22124 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56517
W20260924 00:54:57.801667 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22061: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:54:57.801827 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22061
2026-09-24T00:54:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1836 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:57 dist-test-slave-rtvz krb5kdc[22161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:57 dist-test-slave-rtvz krb5kdc[22161](info): set up 2 sockets
Sep 24 00:54:57 dist-test-slave-rtvz krb5kdc[22161](info): commencing operation
krb5kdc: starting...
W20260924 00:54:58.830833 888 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:54:58 dist-test-slave-rtvz krb5kdc[22161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:54:59.088774 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:54:59.411900 22174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:54:59.412189 22174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:54:59.412324 22174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:54:59.419797 22174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:54:59.419925 22174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:54:59.419994 22174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:54:59.420074 22174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:54:59.420145 22174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:54:59.430341 22174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:54:59.431661 22174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:54:59.433003 22174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:54:59.444877 22180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:59.444940 22179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:54:59.444907 22182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:54:59.446060 22174 server_base.cc:1034] running on GCE node
I20260924 00:54:59.446864 22174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:54:59.448320 22174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:54:59.449555 22174 hybrid_clock.cc:648] HybridClock initialized: now 1790211299449479 us; error 61 us; skew 500 ppm
I20260924 00:54:59.449932 22174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:54:59 dist-test-slave-rtvz krb5kdc[22161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:54:59.453006 22174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:54:59.454598 22174 webserver.cc:460] Webserver started at http://127.0.0.1:45797/ using document root <none> and password file <none>
I20260924 00:54:59.455286 22174 fs_manager.cc:359] Metadata directory not provided
I20260924 00:54:59.455397 22174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:54:59.455691 22174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:54:59.458750 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "8e656e65c02840ddb47b147520fbb3fa"
format_stamp: "Formatted at 2026-09-24 00:54:59 on dist-test-slave-rtvz"
server_key: "e4464e77355ddf2a9f688d53f004db93"
server_key_iv: "dc4215c8c4f8c101b7a21d5bd48c7a4a"
server_key_version: "encryptionkey@0"
I20260924 00:54:59.459507 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "8e656e65c02840ddb47b147520fbb3fa"
format_stamp: "Formatted at 2026-09-24 00:54:59 on dist-test-slave-rtvz"
server_key: "e4464e77355ddf2a9f688d53f004db93"
server_key_iv: "dc4215c8c4f8c101b7a21d5bd48c7a4a"
server_key_version: "encryptionkey@0"
I20260924 00:54:59.465011 22174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 00:54:59.468690 22189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:54:59.470233 22174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:54:59.470402 22174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "8e656e65c02840ddb47b147520fbb3fa"
format_stamp: "Formatted at 2026-09-24 00:54:59 on dist-test-slave-rtvz"
server_key: "e4464e77355ddf2a9f688d53f004db93"
server_key_iv: "dc4215c8c4f8c101b7a21d5bd48c7a4a"
server_key_version: "encryptionkey@0"
I20260924 00:54:59.470616 22174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:54:59.490911 22174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:54:59.495319 22174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:54:59.495608 22174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:54:59.513947 22174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42223
I20260924 00:54:59.513983 22241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42223 every 8 connection(s)
I20260924 00:54:59.515342 22174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:54:59.519138 22242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:54:59.523871 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22174
I20260924 00:54:59.524080 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:54:59.524621 888 external_mini_cluster.cc:1428] Setting key ce6c645d1f77f500b542a779da2ef1b9
I20260924 00:54:59.530344 22242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa: Bootstrap starting.
I20260924 00:54:59.533391 22242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa: Neither blocks nor log segments found. Creating new log.
I20260924 00:54:59.534361 22242 log.cc:824] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa: Log is configured to *not* fsync() on all Append() calls
I20260924 00:54:59.537132 22242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa: No bootstrap required, opened a new log
Sep 24 00:54:59 dist-test-slave-rtvz krb5kdc[22161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:54:59.545979 22242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e656e65c02840ddb47b147520fbb3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260924 00:54:59.546416 22242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:54:59.546535 22242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e656e65c02840ddb47b147520fbb3fa, State: Initialized, Role: FOLLOWER
I20260924 00:54:59.547161 22242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e656e65c02840ddb47b147520fbb3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260924 00:54:59.547290 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:59.526557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:33869 (local address 127.0.0.1:42223)
0924 00:54:59.527311 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:59.527322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:59.527365 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:54:59.529129 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:59.529152 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:59.529597 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:59.529948 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:59.531369 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:59.532679 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:59.534042 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:59.534776 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:59.534970 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:59.539410 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:59.539421 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:59.539436 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:59.539477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:59.542636 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:59.543640 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:59.543647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:59.543657 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:59.543838 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:59.544342 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:59.544345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:59.544347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:59.544935 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:59.545234 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:59.545725 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 00:54:59.546048 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":328,"threads_started":1}
I20260924 00:54:59.547398 22242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:54:59.547520 22242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:54:59.547711 22242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:54:59.550611 22242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e656e65c02840ddb47b147520fbb3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260924 00:54:59.551215 22242 leader_election.cc:302] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e656e65c02840ddb47b147520fbb3fa; no voters:
I20260924 00:54:59.551813 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:54:59.551856 22242 leader_election.cc:288] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:54:59.552135 22247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:54:59.552733 22247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [term 1 LEADER]: Becoming Leader. State: Replica: 8e656e65c02840ddb47b147520fbb3fa, State: Running, Role: LEADER
I20260924 00:54:59.553469 22247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e656e65c02840ddb47b147520fbb3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } }
I20260924 00:54:59.554874 22242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:54:59.560672 22249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e656e65c02840ddb47b147520fbb3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e656e65c02840ddb47b147520fbb3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } } }
I20260924 00:54:59.561158 22249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:59.561691 22248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e656e65c02840ddb47b147520fbb3fa. Latest consensus state: current_term: 1 leader_uuid: "8e656e65c02840ddb47b147520fbb3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e656e65c02840ddb47b147520fbb3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42223 } } }
I20260924 00:54:59.562036 22248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa [sys.catalog]: This master's current role is: LEADER
I20260924 00:54:59.564154 22259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:54:59.572642 22259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:54:59.586418 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:54:59.557268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:42681 (local address 127.0.0.1:42223)
0924 00:54:59.557474 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 00:54:59.557482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:54:59.568411 (+ 10929us) server_negotiation.cc:407] Connection header received
0924 00:54:59.568665 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:54:59.568672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:54:59.568773 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:54:59.568909 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:54:59.570992 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:59.572319 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:59.574437 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:54:59.575156 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:54:59.575311 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:54:59.578638 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:54:59.578646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:54:59.578651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:54:59.578700 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:54:59.582510 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:59.583546 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:59.583552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:59.583556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:59.583769 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:54:59.584448 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:54:59.584457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:54:59.584461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:54:59.584861 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:54:59.585145 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 00:54:59.585830 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 00:54:59.586027 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:54:59.589457 22271 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 00:54:59.589932 22259 catalog_manager.cc:1347] Generated new cluster ID: 8ef99173cbeb4f3294991dae032ccbdc
I20260924 00:54:59.590052 22259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:54:59.602555 22259 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:54:59.604550 22259 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:54:59.627640 22259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e656e65c02840ddb47b147520fbb3fa: Generated new TSK 0
I20260924 00:54:59.628747 22259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:54:59.637781 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22174
I20260924 00:54:59.639546 22237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:54:59.730677 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22174
W20260924 00:54:59.744482 22159 connection.cc:535] client connection to 127.0.0.1:42223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42223): Connection reset by peer (error 104)
2026-09-24T00:54:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:54:59 dist-test-slave-rtvz krb5kdc[22275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:54:59 dist-test-slave-rtvz krb5kdc[22275](info): set up 2 sockets
Sep 24 00:54:59 dist-test-slave-rtvz krb5kdc[22275](info): commencing operation
krb5kdc: starting...
W20260924 00:55:00.756614 888 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:00 dist-test-slave-rtvz krb5kdc[22275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:55:01.018554 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:01.349189 22288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:01.349536 22288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:01.349682 22288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:01.357074 22288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:01.357241 22288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:01.357331 22288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:01.357419 22288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:01.357482 22288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:01.367491 22288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:01.368842 22288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:01.370246 22288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:01.382431 22294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:01.382570 22293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:01.382477 22296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:01.383180 22288 server_base.cc:1034] running on GCE node
I20260924 00:55:01.384034 22288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:01.385675 22288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:01.386922 22288 hybrid_clock.cc:648] HybridClock initialized: now 1790211301386846 us; error 46 us; skew 500 ppm
I20260924 00:55:01.387295 22288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:01 dist-test-slave-rtvz krb5kdc[22275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:01.394354 22288 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:55:01.396054 22288 webserver.cc:460] Webserver started at http://127.0.0.1:40365/ using document root <none> and password file <none>
I20260924 00:55:01.396772 22288 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:01.396895 22288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:01.397248 22288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:01.400374 22288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "bae830b53b5c4ff58c9fea14c19c1ec2"
format_stamp: "Formatted at 2026-09-24 00:55:01 on dist-test-slave-rtvz"
server_key: "77c91ef2e470e873d3214376894c7ba6"
server_key_iv: "7918d7855bb4dfc62c86a34e105e154a"
server_key_version: "encryptionkey@0"
I20260924 00:55:01.401142 22288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "bae830b53b5c4ff58c9fea14c19c1ec2"
format_stamp: "Formatted at 2026-09-24 00:55:01 on dist-test-slave-rtvz"
server_key: "77c91ef2e470e873d3214376894c7ba6"
server_key_iv: "7918d7855bb4dfc62c86a34e105e154a"
server_key_version: "encryptionkey@0"
I20260924 00:55:01.406478 22288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:55:01.410243 22303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:01.411957 22288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:55:01.412143 22288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "bae830b53b5c4ff58c9fea14c19c1ec2"
format_stamp: "Formatted at 2026-09-24 00:55:01 on dist-test-slave-rtvz"
server_key: "77c91ef2e470e873d3214376894c7ba6"
server_key_iv: "7918d7855bb4dfc62c86a34e105e154a"
server_key_version: "encryptionkey@0"
I20260924 00:55:01.412456 22288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:01.429881 22288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:01.435997 22288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:01.436332 22288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:01.457224 22288 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45435
I20260924 00:55:01.457297 22354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45435 every 8 connection(s)
I20260924 00:55:01.459086 22288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:01.464226 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22288
I20260924 00:55:01.464411 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:01.464851 888 external_mini_cluster.cc:1428] Setting key 5de334d8ce5ac259f90b695ca366518c
I20260924 00:55:01.464855 22355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:01.481287 22355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2: Bootstrap starting.
I20260924 00:55:01.481971 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:01.466795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:35207 (local address 127.0.0.1:45435)
0924 00:55:01.467621 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:01.467638 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:01.467684 (+ 46us) server_negotiation.cc:407] Connection header received
0924 00:55:01.471392 (+ 3708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:01.471432 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:01.472160 (+ 728us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:01.472589 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:01.473303 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:01.474788 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:01.476609 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:01.477509 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:01.477675 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:01.479179 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:01.479188 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:01.479205 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:01.479871 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:01.479978 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:01.480009 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:01.480154 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:01.480482 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 00:55:01.480688 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":348,"threads_started":1}
I20260924 00:55:01.484895 22355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:01.486102 22355 log.cc:824] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:01.486236 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:01.489153 22355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2: No bootstrap required, opened a new log
I20260924 00:55:01.497926 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:01.490836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:56651 (local address 127.0.0.1:45435)
0924 00:55:01.491064 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:01.491072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:01.491512 (+ 440us) server_negotiation.cc:407] Connection header received
0924 00:55:01.491713 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:01.491717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:01.491823 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:01.491911 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:01.492521 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:01.493441 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:01.495352 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:01.495834 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:01.496019 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:01.497127 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:01.497131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:01.497135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:01.497162 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:01.497248 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:01.497267 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:01.497364 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:01.497536 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 00:55:01.497581 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:55:01.498903 22355 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae830b53b5c4ff58c9fea14c19c1ec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260924 00:55:01.499308 22355 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:01.499411 22355 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae830b53b5c4ff58c9fea14c19c1ec2, State: Initialized, Role: FOLLOWER
I20260924 00:55:01.499980 22355 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae830b53b5c4ff58c9fea14c19c1ec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260924 00:55:01.500211 22355 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:01.500337 22355 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:01.500486 22355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:01.502729 22355 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae830b53b5c4ff58c9fea14c19c1ec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260924 00:55:01.503260 22355 leader_election.cc:302] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae830b53b5c4ff58c9fea14c19c1ec2; no voters:
I20260924 00:55:01.503859 22355 leader_election.cc:288] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:01.504057 22365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:01.504657 22365 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [term 1 LEADER]: Becoming Leader. State: Replica: bae830b53b5c4ff58c9fea14c19c1ec2, State: Running, Role: LEADER
I20260924 00:55:01.505187 22365 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae830b53b5c4ff58c9fea14c19c1ec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260924 00:55:01.506153 22355 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:01.508083 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae830b53b5c4ff58c9fea14c19c1ec2. Latest consensus state: current_term: 1 leader_uuid: "bae830b53b5c4ff58c9fea14c19c1ec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae830b53b5c4ff58c9fea14c19c1ec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } } }
I20260924 00:55:01.508105 22366 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae830b53b5c4ff58c9fea14c19c1ec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae830b53b5c4ff58c9fea14c19c1ec2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } } }
I20260924 00:55:01.508512 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:01.508512 22366 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:01.509447 22370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:01.514676 22370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:01.524287 22370 catalog_manager.cc:1347] Generated new cluster ID: 9b7b43af7c5c49779e061fd94bf0a94c
I20260924 00:55:01.524415 22370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:01.534094 22370 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:01.536078 22370 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:01.555719 22370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae830b53b5c4ff58c9fea14c19c1ec2: Generated new TSK 0
I20260924 00:55:01.556938 22370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:01.571386 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22288
W20260924 00:55:01.573582 22350 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.100:35207
W20260924 00:55:01.574489 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22288: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:55:01.574659 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22288
W20260924 00:55:01.588685 22273 connection.cc:535] client connection to 127.0.0.1:45435 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45435): Connection reset by peer (error 104)
2026-09-24T00:55:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1844 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:01 dist-test-slave-rtvz krb5kdc[22386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:01 dist-test-slave-rtvz krb5kdc[22386](info): set up 2 sockets
Sep 24 00:55:01 dist-test-slave-rtvz krb5kdc[22386](info): commencing operation
krb5kdc: starting...
W20260924 00:55:02.601441 888 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:02 dist-test-slave-rtvz krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:55:02.863962 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:03.197482 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:03.197777 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:03.197897 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:03.205328 22399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:03.205453 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:03.205524 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:03.205606 22399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:03.205664 22399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:03.215924 22399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:03.217356 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:03.218739 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:03.230700 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:03.231115 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:03.231253 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:03.231860 22399 server_base.cc:1034] running on GCE node
I20260924 00:55:03.232731 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:03.234288 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:03.235523 22399 hybrid_clock.cc:648] HybridClock initialized: now 1790211303235467 us; error 46 us; skew 500 ppm
I20260924 00:55:03.236001 22399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:03 dist-test-slave-rtvz krb5kdc[22386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:03.247177 22399 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:55:03.248884 22399 webserver.cc:460] Webserver started at http://127.0.0.1:36921/ using document root <none> and password file <none>
I20260924 00:55:03.249686 22399 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:03.249811 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:03.250135 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:03.253264 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "f557aaa76ac749adb01367d4eeb05876"
format_stamp: "Formatted at 2026-09-24 00:55:03 on dist-test-slave-rtvz"
server_key: "5658d5971b7e7d58efba40de0ed9043c"
server_key_iv: "874e10383d52d8ab324b5010842d1c89"
server_key_version: "encryptionkey@0"
I20260924 00:55:03.254060 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "f557aaa76ac749adb01367d4eeb05876"
format_stamp: "Formatted at 2026-09-24 00:55:03 on dist-test-slave-rtvz"
server_key: "5658d5971b7e7d58efba40de0ed9043c"
server_key_iv: "874e10383d52d8ab324b5010842d1c89"
server_key_version: "encryptionkey@0"
I20260924 00:55:03.259637 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:55:03.263252 22414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:03.264668 22399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:55:03.264842 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "f557aaa76ac749adb01367d4eeb05876"
format_stamp: "Formatted at 2026-09-24 00:55:03 on dist-test-slave-rtvz"
server_key: "5658d5971b7e7d58efba40de0ed9043c"
server_key_iv: "874e10383d52d8ab324b5010842d1c89"
server_key_version: "encryptionkey@0"
I20260924 00:55:03.265094 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:03.270810 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:03.275146 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:03.275424 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:03.293136 22399 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45789
I20260924 00:55:03.293171 22465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45789 every 8 connection(s)
I20260924 00:55:03.294651 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:03.298455 22466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:03.299424 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22399
I20260924 00:55:03.299640 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:03.300217 888 external_mini_cluster.cc:1428] Setting key 7c72ffbd31545772c5906af424f32e16
I20260924 00:55:03.308225 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:03.302151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:42949 (local address 127.0.0.1:45789)
0924 00:55:03.302984 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:03.302997 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:03.303042 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:55:03.304801 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:03.304830 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:03.305256 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:03.305623 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:03.306688 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:03.306715 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:03.306729 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:03.306743 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:03.306803 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:03.306826 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:03.306986 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:03.307085 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 00:55:03.307297 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":366,"threads_started":1}
I20260924 00:55:03.312274 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:03.312520 22466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876: Bootstrap starting.
I20260924 00:55:03.315735 22466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:03.317101 22466 log.cc:824] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:03.319260 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:03.316721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:40403 (local address 127.0.0.1:45789)
0924 00:55:03.316879 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:03.316883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:03.317366 (+ 483us) server_negotiation.cc:407] Connection header received
0924 00:55:03.317539 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:03.317544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:03.317607 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:03.317730 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:03.318679 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:03.318684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:03.318688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:03.318706 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:03.318749 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:03.318767 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:03.318881 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:03.318929 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 00:55:03.318988 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:55:03.320755 22466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876: No bootstrap required, opened a new log
I20260924 00:55:03.328621 22466 raft_consensus.cc:348] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f557aaa76ac749adb01367d4eeb05876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45789 } }
I20260924 00:55:03.329149 22466 raft_consensus.cc:374] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:03.329298 22466 raft_consensus.cc:729] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f557aaa76ac749adb01367d4eeb05876, State: Initialized, Role: FOLLOWER
I20260924 00:55:03.329895 22466 consensus_queue.cc:260] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f557aaa76ac749adb01367d4eeb05876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45789 } }
I20260924 00:55:03.330132 22466 raft_consensus.cc:388] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:03.330235 22466 raft_consensus.cc:482] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:03.330360 22466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:03.332513 22466 raft_consensus.cc:504] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f557aaa76ac749adb01367d4eeb05876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45789 } }
I20260924 00:55:03.333022 22466 leader_election.cc:302] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f557aaa76ac749adb01367d4eeb05876; no voters:
I20260924 00:55:03.333621 22466 leader_election.cc:288] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:03.333770 22476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:03.334879 22476 raft_consensus.cc:686] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [term 1 LEADER]: Becoming Leader. State: Replica: f557aaa76ac749adb01367d4eeb05876, State: Running, Role: LEADER
I20260924 00:55:03.335510 22476 consensus_queue.cc:237] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f557aaa76ac749adb01367d4eeb05876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45789 } }
I20260924 00:55:03.335884 22466 sys_catalog.cc:564] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:03.338377 22477 sys_catalog.cc:455] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f557aaa76ac749adb01367d4eeb05876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f557aaa76ac749adb01367d4eeb05876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45789 } } }
I20260924 00:55:03.338593 22478 sys_catalog.cc:455] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f557aaa76ac749adb01367d4eeb05876. Latest consensus state: current_term: 1 leader_uuid: "f557aaa76ac749adb01367d4eeb05876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f557aaa76ac749adb01367d4eeb05876" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45789 } } }
I20260924 00:55:03.339088 22477 sys_catalog.cc:458] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:03.339114 22478 sys_catalog.cc:458] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:03.341791 22483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:03.347162 22483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:03.355878 22483 catalog_manager.cc:1347] Generated new cluster ID: 14b2bfed33df45518aa5210a693d0b91
I20260924 00:55:03.356017 22483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:03.381636 22483 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:03.382208 22483 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:03.392589 22483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f557aaa76ac749adb01367d4eeb05876: Generated new TSK 0
I20260924 00:55:03.393775 22483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:03.401325 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22399
W20260924 00:55:03.402446 22461 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.100:42949
W20260924 00:55:03.403160 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22399: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:55:03.403329 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22399
2026-09-24T00:55:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1828 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:03 dist-test-slave-rtvz krb5kdc[22497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:03 dist-test-slave-rtvz krb5kdc[22497](info): set up 2 sockets
Sep 24 00:55:03 dist-test-slave-rtvz krb5kdc[22497](info): commencing operation
krb5kdc: starting...
W20260924 00:55:04.444782 888 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:04 dist-test-slave-rtvz krb5kdc[22497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:55:04.709745 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:05.040306 22510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:05.040640 22510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:05.040786 22510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:05.048990 22510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:05.049139 22510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:05.049243 22510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:05.049328 22510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:05.049391 22510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:05.059628 22510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:05.060951 22510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:05.062331 22510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:05.073927 22515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:05.074162 22518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:05.074169 22516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:05.075172 22510 server_base.cc:1034] running on GCE node
I20260924 00:55:05.076048 22510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:05.078065 22510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:05.079299 22510 hybrid_clock.cc:648] HybridClock initialized: now 1790211305079245 us; error 48 us; skew 500 ppm
I20260924 00:55:05.079668 22510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:05 dist-test-slave-rtvz krb5kdc[22497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:05.083133 22510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:55:05.084784 22510 webserver.cc:460] Webserver started at http://127.0.0.1:34171/ using document root <none> and password file <none>
I20260924 00:55:05.085558 22510 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:05.085683 22510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:05.085987 22510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:05.089155 22510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "0aaa208150934220a747e177b5c9a953"
format_stamp: "Formatted at 2026-09-24 00:55:05 on dist-test-slave-rtvz"
server_key: "91d030fd7c8afe7b93d2574da02548ba"
server_key_iv: "fce31f06e0837ecdb6afaef0f86ebd9f"
server_key_version: "encryptionkey@0"
I20260924 00:55:05.090006 22510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "0aaa208150934220a747e177b5c9a953"
format_stamp: "Formatted at 2026-09-24 00:55:05 on dist-test-slave-rtvz"
server_key: "91d030fd7c8afe7b93d2574da02548ba"
server_key_iv: "fce31f06e0837ecdb6afaef0f86ebd9f"
server_key_version: "encryptionkey@0"
I20260924 00:55:05.095517 22510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:55:05.099179 22525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:05.100597 22510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 00:55:05.100759 22510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "0aaa208150934220a747e177b5c9a953"
format_stamp: "Formatted at 2026-09-24 00:55:05 on dist-test-slave-rtvz"
server_key: "91d030fd7c8afe7b93d2574da02548ba"
server_key_iv: "fce31f06e0837ecdb6afaef0f86ebd9f"
server_key_version: "encryptionkey@0"
I20260924 00:55:05.100942 22510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:05.116158 22510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:05.120606 22510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:05.120884 22510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:05.139246 22510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41055
I20260924 00:55:05.139286 22577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41055 every 8 connection(s)
I20260924 00:55:05.140705 22510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:05.144496 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22510
I20260924 00:55:05.144691 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:05.144938 22578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:05.145300 888 external_mini_cluster.cc:1428] Setting key bbfa1ad756a0d451b9f87d678a0f6290
I20260924 00:55:05.160058 22578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953: Bootstrap starting.
Sep 24 00:55:05 dist-test-slave-rtvz krb5kdc[22497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:55:05.163301 22578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:05.164340 22578 log.cc:824] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:05.167208 22578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953: No bootstrap required, opened a new log
I20260924 00:55:05.169384 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:05.147268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:42711 (local address 127.0.0.1:41055)
0924 00:55:05.148096 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:05.148108 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:05.148154 (+ 46us) server_negotiation.cc:407] Connection header received
0924 00:55:05.149908 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:05.149933 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:05.150376 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:05.150761 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:05.152725 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:05.154141 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:05.155871 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:05.156807 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:05.156976 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:05.161780 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:05.161789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:05.161804 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:05.161843 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:05.164782 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:05.165689 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:05.165696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:05.165703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:05.165862 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:05.166304 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:05.166308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:05.166310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:05.166824 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:05.167148 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:05.168037 (+ 889us) server_negotiation.cc:299] Negotiation successful
0924 00:55:05.168295 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":343,"threads_started":1}
I20260924 00:55:05.173749 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:05.176026 22578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaa208150934220a747e177b5c9a953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260924 00:55:05.176558 22578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:05.176664 22578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aaa208150934220a747e177b5c9a953, State: Initialized, Role: FOLLOWER
I20260924 00:55:05.177413 22578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaa208150934220a747e177b5c9a953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260924 00:55:05.177731 22578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:05.177889 22578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:05.178059 22578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:05.180882 22578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaa208150934220a747e177b5c9a953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260924 00:55:05.181432 22578 leader_election.cc:302] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aaa208150934220a747e177b5c9a953; no voters:
I20260924 00:55:05.182092 22578 leader_election.cc:288] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:05.182235 22588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:05.184332 22588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [term 1 LEADER]: Becoming Leader. State: Replica: 0aaa208150934220a747e177b5c9a953, State: Running, Role: LEADER
I20260924 00:55:05.184490 22578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:05.184971 22588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaa208150934220a747e177b5c9a953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } }
I20260924 00:55:05.188076 22589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aaa208150934220a747e177b5c9a953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaa208150934220a747e177b5c9a953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } } }
I20260924 00:55:05.188489 22589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:05.197048 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:05.179034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:38805 (local address 127.0.0.1:41055)
0924 00:55:05.179246 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:05.179252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:05.179768 (+ 516us) server_negotiation.cc:407] Connection header received
0924 00:55:05.179917 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:05.179923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:05.180112 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:05.180227 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:05.181661 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:05.182695 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:05.184110 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:05.184768 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:05.184917 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:05.187733 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:05.187741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:05.187748 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:05.187795 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:05.191982 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:05.193298 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:05.193305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:05.193309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:05.193525 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:05.194162 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:05.194168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:05.194172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:05.194625 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:05.194932 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:05.196403 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0924 00:55:05.196638 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:55:05.189935 22590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aaa208150934220a747e177b5c9a953. Latest consensus state: current_term: 1 leader_uuid: "0aaa208150934220a747e177b5c9a953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aaa208150934220a747e177b5c9a953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41055 } } }
I20260924 00:55:05.198033 22590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:05.202059 22596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:05.207855 22596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:05.217586 22596 catalog_manager.cc:1347] Generated new cluster ID: 894a48815a934843b13ebdcd9a2b5835
I20260924 00:55:05.217705 22596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:05.230096 22596 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:05.231455 22596 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:05.238341 22596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aaa208150934220a747e177b5c9a953: Generated new TSK 0
I20260924 00:55:05.239289 22596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:05.246395 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22510
I20260924 00:55:05.248183 22573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:05.337092 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22510
W20260924 00:55:05.350832 22495 connection.cc:535] client connection to 127.0.0.1:41055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41055): Connection reset by peer (error 104)
2026-09-24T00:55:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:05 dist-test-slave-rtvz krb5kdc[22610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:05 dist-test-slave-rtvz krb5kdc[22610](info): set up 2 sockets
Sep 24 00:55:05 dist-test-slave-rtvz krb5kdc[22610](info): commencing operation
krb5kdc: starting...
W20260924 00:55:06.368664 888 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:06 dist-test-slave-rtvz krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:55:06.629515 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:06.958086 22623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:06.958403 22623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:06.958498 22623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:06.965855 22623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:06.965977 22623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:06.966023 22623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:06.966075 22623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:06.966125 22623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:06.976557 22623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:06.977941 22623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:06.979179 22623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:06.991153 22631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:06.991189 22628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:06.991159 22629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:06.992190 22623 server_base.cc:1034] running on GCE node
I20260924 00:55:06.992975 22623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:06.994525 22623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:06.995728 22623 hybrid_clock.cc:648] HybridClock initialized: now 1790211306995685 us; error 34 us; skew 500 ppm
I20260924 00:55:06.996050 22623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:06 dist-test-slave-rtvz krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:06.998999 22623 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:55:07.000504 22623 webserver.cc:460] Webserver started at http://127.0.0.1:38397/ using document root <none> and password file <none>
I20260924 00:55:07.001169 22623 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:07.001319 22623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:07.001585 22623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:07.004528 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "dd437a3512204db69c585968eb8bd41a"
format_stamp: "Formatted at 2026-09-24 00:55:06 on dist-test-slave-rtvz"
server_key: "e707634571138bc4c7b7a43d71b5651b"
server_key_iv: "1b15066c73988a285773e1e220ca3346"
server_key_version: "encryptionkey@0"
I20260924 00:55:07.005283 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "dd437a3512204db69c585968eb8bd41a"
format_stamp: "Formatted at 2026-09-24 00:55:06 on dist-test-slave-rtvz"
server_key: "e707634571138bc4c7b7a43d71b5651b"
server_key_iv: "1b15066c73988a285773e1e220ca3346"
server_key_version: "encryptionkey@0"
I20260924 00:55:07.010565 22623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:55:07.014076 22638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:07.015443 22623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:55:07.015607 22623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "dd437a3512204db69c585968eb8bd41a"
format_stamp: "Formatted at 2026-09-24 00:55:06 on dist-test-slave-rtvz"
server_key: "e707634571138bc4c7b7a43d71b5651b"
server_key_iv: "1b15066c73988a285773e1e220ca3346"
server_key_version: "encryptionkey@0"
I20260924 00:55:07.015787 22623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:07.039316 22623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:07.043669 22623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:07.043946 22623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:07.062084 22623 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44159
I20260924 00:55:07.062129 22689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44159 every 8 connection(s)
I20260924 00:55:07.063539 22623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:07.067655 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:07.073881 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22623
I20260924 00:55:07.074061 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:07.074453 888 external_mini_cluster.cc:1428] Setting key cd2d496f5b39a1eeed9d8e175b9f4f31
I20260924 00:55:07.078948 22690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a: Bootstrap starting.
I20260924 00:55:07.082769 22690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:07.084128 22690 log.cc:824] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:07.087805 22690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a: No bootstrap required, opened a new log
I20260924 00:55:07.090049 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:07.076470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:59247 (local address 127.0.0.1:44159)
0924 00:55:07.077388 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:07.077401 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:07.077444 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:55:07.079257 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:07.079285 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:07.079762 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:07.080143 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:07.080836 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:07.082503 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:07.084639 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:07.085841 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:07.085984 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:07.087407 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:07.087414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:07.087437 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:07.088015 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:07.088111 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:07.088139 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:07.088258 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:07.088577 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 00:55:07.088767 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":346,"threads_started":1}
I20260924 00:55:07.094120 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:07.096487 22690 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd437a3512204db69c585968eb8bd41a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44159 } }
I20260924 00:55:07.096930 22690 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:07.097044 22690 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd437a3512204db69c585968eb8bd41a, State: Initialized, Role: FOLLOWER
I20260924 00:55:07.097743 22690 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd437a3512204db69c585968eb8bd41a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44159 } }
I20260924 00:55:07.097972 22690 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:07.098099 22690 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:07.098268 22690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:07.101140 22690 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd437a3512204db69c585968eb8bd41a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44159 } }
I20260924 00:55:07.101766 22690 leader_election.cc:302] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd437a3512204db69c585968eb8bd41a; no voters:
I20260924 00:55:07.102401 22690 leader_election.cc:288] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:07.102494 22700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:07.103663 22700 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [term 1 LEADER]: Becoming Leader. State: Replica: dd437a3512204db69c585968eb8bd41a, State: Running, Role: LEADER
I20260924 00:55:07.104194 22700 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd437a3512204db69c585968eb8bd41a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44159 } }
I20260924 00:55:07.104780 22690 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:07.105903 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:07.098389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:37783 (local address 127.0.0.1:44159)
0924 00:55:07.098579 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:07.098585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:07.099152 (+ 567us) server_negotiation.cc:407] Connection header received
0924 00:55:07.099342 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:07.099346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:07.099414 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:07.099493 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:07.100038 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:07.100810 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:07.102633 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:07.103386 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:07.103500 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:07.105034 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:07.105039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:07.105044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:07.105067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:07.105138 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:07.105159 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:07.105295 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:07.105482 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 00:55:07.105535 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:55:07.107784 22701 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd437a3512204db69c585968eb8bd41a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd437a3512204db69c585968eb8bd41a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44159 } } }
I20260924 00:55:07.108093 22701 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:07.108909 22705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:07.108740 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd437a3512204db69c585968eb8bd41a. Latest consensus state: current_term: 1 leader_uuid: "dd437a3512204db69c585968eb8bd41a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd437a3512204db69c585968eb8bd41a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44159 } } }
I20260924 00:55:07.109068 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:07.115036 22705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:07.124775 22705 catalog_manager.cc:1347] Generated new cluster ID: 5a0d068e5dec450697e0e9fb8b2ec4ae
I20260924 00:55:07.124912 22705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:07.143857 22705 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:07.145237 22705 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:07.158684 22705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd437a3512204db69c585968eb8bd41a: Generated new TSK 0
I20260924 00:55:07.159622 22705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:07.172801 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22623
W20260924 00:55:07.174477 22685 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.100:59247
W20260924 00:55:07.175376 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22623: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:55:07.175549 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22623
W20260924 00:55:07.189572 22608 connection.cc:535] client connection to 127.0.0.1:44159 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44159): Connection reset by peer (error 104)
2026-09-24T00:55:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1838 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:07 dist-test-slave-rtvz krb5kdc[22721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:07 dist-test-slave-rtvz krb5kdc[22721](info): set up 2 sockets
Sep 24 00:55:07 dist-test-slave-rtvz krb5kdc[22721](info): commencing operation
krb5kdc: starting...
W20260924 00:55:08.196861 888 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:08 dist-test-slave-rtvz krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:55:08.452699 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:08.783217 22734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:08.783563 22734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:08.783877 22734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:08.791555 22734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:08.791692 22734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:08.791750 22734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:08.791838 22734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:08.791895 22734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:08.802208 22734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:08.803553 22734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:08.804940 22734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:08.817314 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:08.817307 22739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:08.817286 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:08.817996 22734 server_base.cc:1034] running on GCE node
I20260924 00:55:08.818836 22734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:08.820449 22734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:08.821673 22734 hybrid_clock.cc:648] HybridClock initialized: now 1790211308821609 us; error 60 us; skew 500 ppm
I20260924 00:55:08.822089 22734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:08 dist-test-slave-rtvz krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:08.829814 22734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:55:08.831502 22734 webserver.cc:460] Webserver started at http://127.0.0.1:37629/ using document root <none> and password file <none>
I20260924 00:55:08.832222 22734 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:08.832345 22734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:08.832630 22734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:08.835798 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "df49b37e1a9e4775819e366459a9a47c"
format_stamp: "Formatted at 2026-09-24 00:55:08 on dist-test-slave-rtvz"
server_key: "b506a4199c02ac38d1ec3f9eb0ea3f24"
server_key_iv: "de258a44eb2e243c1b74edee3a1807fb"
server_key_version: "encryptionkey@0"
I20260924 00:55:08.836558 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "df49b37e1a9e4775819e366459a9a47c"
format_stamp: "Formatted at 2026-09-24 00:55:08 on dist-test-slave-rtvz"
server_key: "b506a4199c02ac38d1ec3f9eb0ea3f24"
server_key_iv: "de258a44eb2e243c1b74edee3a1807fb"
server_key_version: "encryptionkey@0"
I20260924 00:55:08.841962 22734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 00:55:08.845541 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:08.847036 22734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:55:08.847260 22734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "df49b37e1a9e4775819e366459a9a47c"
format_stamp: "Formatted at 2026-09-24 00:55:08 on dist-test-slave-rtvz"
server_key: "b506a4199c02ac38d1ec3f9eb0ea3f24"
server_key_iv: "de258a44eb2e243c1b74edee3a1807fb"
server_key_version: "encryptionkey@0"
I20260924 00:55:08.847461 22734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:08.853298 22734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:08.857693 22734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:08.858012 22734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:08.875869 22734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42473
I20260924 00:55:08.875975 22800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42473 every 8 connection(s)
I20260924 00:55:08.877374 22734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:08.878022 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22734
I20260924 00:55:08.878219 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:08.878751 888 external_mini_cluster.cc:1428] Setting key 9f2c8e33b6288612fbc615b49ac0150e
I20260924 00:55:08.881875 22801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:08.886603 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:08.880557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:45847 (local address 127.0.0.1:42473)
0924 00:55:08.881358 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:08.881370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:08.881412 (+ 42us) server_negotiation.cc:407] Connection header received
0924 00:55:08.883085 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:08.883108 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:08.883464 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:08.883795 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:08.884890 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:08.884897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:08.884910 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:08.884925 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:08.885011 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:08.885032 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:08.885306 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:08.885410 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 00:55:08.885643 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":308,"threads_started":1}
I20260924 00:55:08.890280 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:08.894537 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c: Bootstrap starting.
I20260924 00:55:08.898018 22801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:08.898093 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:08.895580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.100:55851 (local address 127.0.0.1:42473)
0924 00:55:08.895778 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:08.895785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:08.896287 (+ 502us) server_negotiation.cc:407] Connection header received
0924 00:55:08.896448 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:08.896451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:08.896501 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:08.896596 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:08.897473 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:08.897478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:08.897483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:08.897499 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:08.897540 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:08.897555 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:08.897676 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:08.897756 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 00:55:08.897819 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 00:55:08.899288 22801 log.cc:824] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:08.902295 22801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c: No bootstrap required, opened a new log
I20260924 00:55:08.909970 22801 raft_consensus.cc:348] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df49b37e1a9e4775819e366459a9a47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42473 } }
I20260924 00:55:08.910356 22801 raft_consensus.cc:374] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:08.910434 22801 raft_consensus.cc:729] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df49b37e1a9e4775819e366459a9a47c, State: Initialized, Role: FOLLOWER
I20260924 00:55:08.911005 22801 consensus_queue.cc:260] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df49b37e1a9e4775819e366459a9a47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42473 } }
I20260924 00:55:08.911207 22801 raft_consensus.cc:388] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:08.911319 22801 raft_consensus.cc:482] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:08.911448 22801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:08.913674 22801 raft_consensus.cc:504] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df49b37e1a9e4775819e366459a9a47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42473 } }
I20260924 00:55:08.914247 22801 leader_election.cc:302] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df49b37e1a9e4775819e366459a9a47c; no voters:
I20260924 00:55:08.914912 22801 leader_election.cc:288] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:08.915125 22811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:08.915737 22811 raft_consensus.cc:686] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [term 1 LEADER]: Becoming Leader. State: Replica: df49b37e1a9e4775819e366459a9a47c, State: Running, Role: LEADER
I20260924 00:55:08.916253 22811 consensus_queue.cc:237] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df49b37e1a9e4775819e366459a9a47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42473 } }
I20260924 00:55:08.916940 22801 sys_catalog.cc:564] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:08.918931 22813 sys_catalog.cc:455] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [sys.catalog]: SysCatalogTable state changed. Reason: New leader df49b37e1a9e4775819e366459a9a47c. Latest consensus state: current_term: 1 leader_uuid: "df49b37e1a9e4775819e366459a9a47c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df49b37e1a9e4775819e366459a9a47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42473 } } }
I20260924 00:55:08.919245 22813 sys_catalog.cc:458] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:08.920070 22818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:08.919914 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df49b37e1a9e4775819e366459a9a47c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df49b37e1a9e4775819e366459a9a47c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42473 } } }
I20260924 00:55:08.920222 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:08.925518 22818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:08.935129 22818 catalog_manager.cc:1347] Generated new cluster ID: fbbb10d02e3a4d6c81370be5bb65d0a9
I20260924 00:55:08.935262 22818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:08.974078 22818 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:08.974560 22818 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:08.983433 22818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df49b37e1a9e4775819e366459a9a47c: Generated new TSK 0
I20260924 00:55:08.984360 22818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:08.990679 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22734
W20260924 00:55:08.991868 22796 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.100:45847
W20260924 00:55:08.992630 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22734: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:55:08.992781 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22734
2026-09-24T00:55:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:09 dist-test-slave-rtvz krb5kdc[22832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:09 dist-test-slave-rtvz krb5kdc[22832](info): set up 2 sockets
Sep 24 00:55:09 dist-test-slave-rtvz krb5kdc[22832](info): commencing operation
krb5kdc: starting...
W20260924 00:55:10.020710 888 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:10 dist-test-slave-rtvz krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:55:10.272693 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37201
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:37201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:10.602880 22845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:10.603195 22845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:10.603303 22845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:10.610581 22845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:10.610697 22845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:10.610751 22845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:10.610808 22845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:10.610849 22845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:10.621013 22845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:37201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:37201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:10.622341 22845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:10.623620 22845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:10.635226 22850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:10.635337 22853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:10.635337 22851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:10.636237 22845 server_base.cc:1034] running on GCE node
I20260924 00:55:10.637071 22845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:10.638520 22845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:10.639719 22845 hybrid_clock.cc:648] HybridClock initialized: now 1790211310639661 us; error 42 us; skew 500 ppm
I20260924 00:55:10.640113 22845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:10 dist-test-slave-rtvz krb5kdc[22832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:10.643214 22845 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:55:10.644672 22845 webserver.cc:460] Webserver started at http://127.0.222.62:35111/ using document root <none> and password file <none>
I20260924 00:55:10.645390 22845 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:10.645499 22845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:10.645799 22845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:10.648749 22845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "0bca74ecabad420d854991d966e1757a"
format_stamp: "Formatted at 2026-09-24 00:55:10 on dist-test-slave-rtvz"
server_key: "75a28fa7d53e3d6bb5182218e92eb32a"
server_key_iv: "69051ac7282e986455ed60400e6da967"
server_key_version: "encryptionkey@0"
I20260924 00:55:10.649562 22845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "0bca74ecabad420d854991d966e1757a"
format_stamp: "Formatted at 2026-09-24 00:55:10 on dist-test-slave-rtvz"
server_key: "75a28fa7d53e3d6bb5182218e92eb32a"
server_key_iv: "69051ac7282e986455ed60400e6da967"
server_key_version: "encryptionkey@0"
I20260924 00:55:10.654738 22845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 00:55:10.658356 22860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:10.659796 22845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 00:55:10.659973 22845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "0bca74ecabad420d854991d966e1757a"
format_stamp: "Formatted at 2026-09-24 00:55:10 on dist-test-slave-rtvz"
server_key: "75a28fa7d53e3d6bb5182218e92eb32a"
server_key_iv: "69051ac7282e986455ed60400e6da967"
server_key_version: "encryptionkey@0"
I20260924 00:55:10.660174 22845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:10.685674 22845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:10.690112 22845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:10.690379 22845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:10.708606 22845 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:37201
I20260924 00:55:10.708637 22911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:37201 every 8 connection(s)
I20260924 00:55:10.710029 22845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:10.713729 22912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:10.717672 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22845
I20260924 00:55:10.717855 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:10.718382 888 external_mini_cluster.cc:1428] Setting key 5f88a58dff1417419f320832c3049900
I20260924 00:55:10.726709 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a: Bootstrap starting.
I20260924 00:55:10.730557 22912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:10.731892 22912 log.cc:824] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a: Log is configured to *not* fsync() on all Append() calls
Sep 24 00:55:10 dist-test-slave-rtvz krb5kdc[22832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:55:10.735234 22912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a: No bootstrap required, opened a new log
I20260924 00:55:10.741709 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:10.720194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57949 (local address 127.0.222.62:37201)
0924 00:55:10.720924 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:10.720943 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:10.720984 (+ 41us) server_negotiation.cc:407] Connection header received
0924 00:55:10.722562 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:10.722585 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:10.723019 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:10.723336 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:10.725119 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:10.726572 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:10.728305 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:10.729000 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:10.729192 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:10.734178 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:10.734186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:10.734201 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:10.734240 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:10.737339 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:10.738235 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:10.738242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:10.738248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:10.738407 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:10.738842 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:10.738845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:10.738847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:10.739422 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:10.739655 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:10.740203 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 00:55:10.740529 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":333,"threads_started":1}
I20260924 00:55:10.744388 22912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bca74ecabad420d854991d966e1757a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37201 } }
I20260924 00:55:10.744841 22912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:10.744930 22912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bca74ecabad420d854991d966e1757a, State: Initialized, Role: FOLLOWER
I20260924 00:55:10.745574 22912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bca74ecabad420d854991d966e1757a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37201 } }
I20260924 00:55:10.745832 22912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:10.745980 22912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:10.746019 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:10.746140 22912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:10.748270 22912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bca74ecabad420d854991d966e1757a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37201 } }
I20260924 00:55:10.748795 22912 leader_election.cc:302] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bca74ecabad420d854991d966e1757a; no voters:
I20260924 00:55:10.749408 22912 leader_election.cc:288] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:10.749503 22921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:10.751709 22912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:10.753888 22921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [term 1 LEADER]: Becoming Leader. State: Replica: 0bca74ecabad420d854991d966e1757a, State: Running, Role: LEADER
I20260924 00:55:10.754547 22921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bca74ecabad420d854991d966e1757a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37201 } }
I20260924 00:55:10.757311 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bca74ecabad420d854991d966e1757a. Latest consensus state: current_term: 1 leader_uuid: "0bca74ecabad420d854991d966e1757a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bca74ecabad420d854991d966e1757a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37201 } } }
I20260924 00:55:10.757686 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:10.760011 22922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bca74ecabad420d854991d966e1757a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bca74ecabad420d854991d966e1757a" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 37201 } } }
I20260924 00:55:10.761049 22922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:10.761829 22931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:10.769151 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:10.750433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39109 (local address 127.0.222.62:37201)
0924 00:55:10.750585 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:10.750591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:10.751313 (+ 722us) server_negotiation.cc:407] Connection header received
0924 00:55:10.751577 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:10.751583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:10.751651 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:10.751737 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:10.753179 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:10.754610 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:10.756391 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:10.757067 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:10.757223 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:10.760028 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:10.760036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:10.760041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:10.760082 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:10.764469 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:10.765768 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:10.765773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:10.765776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:10.765926 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:10.766541 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:10.766546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:10.766549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:10.766968 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:10.768385 (+ 1417us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:10.768568 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 00:55:10.768744 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:55:10.772991 22931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:10.783365 22931 catalog_manager.cc:1347] Generated new cluster ID: 9fa60a67c5d644f88dcf69e2b0ca9812
I20260924 00:55:10.783484 22931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:10.813737 22931 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:10.815840 22931 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:10.827732 22931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bca74ecabad420d854991d966e1757a: Generated new TSK 0
I20260924 00:55:10.828838 22931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:10.842496 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22845
I20260924 00:55:10.843770 22907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:10.936937 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22845
W20260924 00:55:10.952132 22830 connection.cc:535] client connection to 127.0.222.62:37201 recv error: Network error: failed to read from TLS socket (remote: 127.0.222.62:37201): Connection reset by peer (error 104)
2026-09-24T00:55:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1944 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:11 dist-test-slave-rtvz krb5kdc[22944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:11 dist-test-slave-rtvz krb5kdc[22944](info): set up 2 sockets
Sep 24 00:55:11 dist-test-slave-rtvz krb5kdc[22944](info): commencing operation
krb5kdc: starting...
W20260924 00:55:11.960624 888 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:12 dist-test-slave-rtvz krb5kdc[22944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:55:12.217239 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39931
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:45811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:39931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:12.542030 22957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:12.542338 22957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:12.542483 22957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:12.549714 22957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:12.549846 22957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:12.549927 22957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:12.550014 22957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:12.550071 22957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:12.560664 22957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:45811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:39931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:39931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:12.562116 22957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:12.563488 22957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:12.575110 22962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:12.575053 22965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:12.575140 22963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:12.576188 22957 server_base.cc:1034] running on GCE node
I20260924 00:55:12.576960 22957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:12.578442 22957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:12.579638 22957 hybrid_clock.cc:648] HybridClock initialized: now 1790211312579592 us; error 38 us; skew 500 ppm
I20260924 00:55:12.579969 22957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:12 dist-test-slave-rtvz krb5kdc[22944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:12.582911 22957 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:55:12.584275 22957 webserver.cc:460] Webserver started at http://127.0.222.62:41779/ using document root <none> and password file <none>
I20260924 00:55:12.584910 22957 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:12.585004 22957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:12.585314 22957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:12.588181 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971"
format_stamp: "Formatted at 2026-09-24 00:55:12 on dist-test-slave-rtvz"
server_key: "8381edbb481643eab0592e3ada2d4ae3"
server_key_iv: "e0e2c58dcbfab37f357ce03836a143c8"
server_key_version: "encryptionkey@0"
I20260924 00:55:12.588889 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971"
format_stamp: "Formatted at 2026-09-24 00:55:12 on dist-test-slave-rtvz"
server_key: "8381edbb481643eab0592e3ada2d4ae3"
server_key_iv: "e0e2c58dcbfab37f357ce03836a143c8"
server_key_version: "encryptionkey@0"
I20260924 00:55:12.593977 22957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:55:12.597378 22972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:12.598749 22957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:55:12.598891 22957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971"
format_stamp: "Formatted at 2026-09-24 00:55:12 on dist-test-slave-rtvz"
server_key: "8381edbb481643eab0592e3ada2d4ae3"
server_key_iv: "e0e2c58dcbfab37f357ce03836a143c8"
server_key_version: "encryptionkey@0"
I20260924 00:55:12.599072 22957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:12.613510 22957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:12.617640 22957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:12.617892 22957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:12.634610 22957 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:39931
I20260924 00:55:12.634657 23023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:39931 every 8 connection(s)
I20260924 00:55:12.635967 22957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:12.639846 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:12.641330 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 22957
I20260924 00:55:12.641520 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:12.642076 888 external_mini_cluster.cc:1428] Setting key a9abc791623c69c09a730410f00760c9
I20260924 00:55:12.652922 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971: Bootstrap starting.
I20260924 00:55:12.654809 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:12.644061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43157 (local address 127.0.222.62:39931)
0924 00:55:12.644815 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:12.644827 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:12.644870 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:55:12.646495 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:12.646519 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:12.646967 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:12.647345 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:12.647862 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:12.649099 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:12.650403 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:12.651119 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:12.651280 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:12.652548 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:12.652567 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:12.652582 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:12.653133 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:12.653241 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:12.653276 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:12.653423 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:12.653692 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 00:55:12.653859 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":334,"threads_started":1}
I20260924 00:55:12.656172 23024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:12.657298 23024 log.cc:824] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:12.658528 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:12.660029 23024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971: No bootstrap required, opened a new log
I20260924 00:55:12.667690 23024 raft_consensus.cc:348] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39931 } }
I20260924 00:55:12.668107 23024 raft_consensus.cc:374] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:12.668215 23024 raft_consensus.cc:729] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54d8f1f8df74e3ab0b0b6e7058c5971, State: Initialized, Role: FOLLOWER
I20260924 00:55:12.668641 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:12.662701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37607 (local address 127.0.222.62:39931)
0924 00:55:12.662869 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:12.662876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:12.663277 (+ 401us) server_negotiation.cc:407] Connection header received
0924 00:55:12.663454 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:12.663458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:12.663525 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:12.663605 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:12.664087 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:12.664876 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:12.666248 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:12.666708 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:12.666817 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:12.667897 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:12.667902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:12.667905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:12.667922 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:12.667980 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:12.667995 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:12.668085 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:12.668286 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 00:55:12.668339 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:55:12.668937 23024 consensus_queue.cc:260] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39931 } }
I20260924 00:55:12.669212 23024 raft_consensus.cc:388] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:12.669358 23024 raft_consensus.cc:482] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:12.669570 23024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:12.671912 23024 raft_consensus.cc:504] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39931 } }
I20260924 00:55:12.672381 23024 leader_election.cc:302] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b54d8f1f8df74e3ab0b0b6e7058c5971; no voters:
I20260924 00:55:12.672924 23024 leader_election.cc:288] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:12.673059 23034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:12.673907 23034 raft_consensus.cc:686] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [term 1 LEADER]: Becoming Leader. State: Replica: b54d8f1f8df74e3ab0b0b6e7058c5971, State: Running, Role: LEADER
I20260924 00:55:12.674809 23034 consensus_queue.cc:237] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39931 } }
I20260924 00:55:12.675230 23024 sys_catalog.cc:564] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:12.677722 23036 sys_catalog.cc:455] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b54d8f1f8df74e3ab0b0b6e7058c5971. Latest consensus state: current_term: 1 leader_uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39931 } } }
I20260924 00:55:12.678103 23036 sys_catalog.cc:458] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:12.681624 23035 sys_catalog.cc:455] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54d8f1f8df74e3ab0b0b6e7058c5971" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 39931 } } }
I20260924 00:55:12.681980 23035 sys_catalog.cc:458] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:12.682530 23043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:12.688138 23043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:12.697585 23043 catalog_manager.cc:1347] Generated new cluster ID: a3226e292f554923a6500aa7c1591559
I20260924 00:55:12.697714 23043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:12.719274 23043 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:12.720755 23043 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:12.732923 23043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b54d8f1f8df74e3ab0b0b6e7058c5971: Generated new TSK 0
I20260924 00:55:12.733914 23043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:12.748688 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22957
W20260924 00:55:12.750006 23019 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43157
W20260924 00:55:12.750747 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 22957: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:55:12.750905 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 22957
W20260924 00:55:12.764647 22942 connection.cc:535] client connection to 127.0.222.62:39931 recv error: Network error: failed to read from TLS socket (remote: 127.0.222.62:39931): Connection reset by peer (error 104)
2026-09-24T00:55:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1812 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:12 dist-test-slave-rtvz krb5kdc[23055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:12 dist-test-slave-rtvz krb5kdc[23055](info): set up 2 sockets
Sep 24 00:55:12 dist-test-slave-rtvz krb5kdc[23055](info): commencing operation
krb5kdc: starting...
W20260924 00:55:13.764859 888 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:13 dist-test-slave-rtvz krb5kdc[23055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:55:14.021741 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41531
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:14.350308 23068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:14.350594 23068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:14.350708 23068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:14.357909 23068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:14.358031 23068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:14.358089 23068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:14.358170 23068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:14.358227 23068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:14.368192 23068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:14.369510 23068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:14.370792 23068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:14.381935 23073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:14.381978 23076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:14.382007 23074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:14.383001 23068 server_base.cc:1034] running on GCE node
I20260924 00:55:14.383767 23068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:14.385140 23068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:14.386358 23068 hybrid_clock.cc:648] HybridClock initialized: now 1790211314386305 us; error 45 us; skew 500 ppm
I20260924 00:55:14.386686 23068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:14 dist-test-slave-rtvz krb5kdc[23055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:14.394949 23068 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:55:14.396492 23068 webserver.cc:460] Webserver started at http://127.0.222.62:39919/ using document root <none> and password file <none>
I20260924 00:55:14.397181 23068 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:14.397336 23068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:14.397630 23068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:14.400543 23068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "07dfd0eafccd498e83ee47291fb44538"
format_stamp: "Formatted at 2026-09-24 00:55:14 on dist-test-slave-rtvz"
server_key: "f880b980e59a9de325276f50bbb822b8"
server_key_iv: "092a09e2e01d8e5f685fdb1ce633a0a5"
server_key_version: "encryptionkey@0"
I20260924 00:55:14.401314 23068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "07dfd0eafccd498e83ee47291fb44538"
format_stamp: "Formatted at 2026-09-24 00:55:14 on dist-test-slave-rtvz"
server_key: "f880b980e59a9de325276f50bbb822b8"
server_key_iv: "092a09e2e01d8e5f685fdb1ce633a0a5"
server_key_version: "encryptionkey@0"
I20260924 00:55:14.406507 23068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 00:55:14.409863 23083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:14.411242 23068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 00:55:14.411401 23068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "07dfd0eafccd498e83ee47291fb44538"
format_stamp: "Formatted at 2026-09-24 00:55:14 on dist-test-slave-rtvz"
server_key: "f880b980e59a9de325276f50bbb822b8"
server_key_iv: "092a09e2e01d8e5f685fdb1ce633a0a5"
server_key_version: "encryptionkey@0"
I20260924 00:55:14.411582 23068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:14.417241 23068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:14.421391 23068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:14.421661 23068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:14.438119 23068 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41531
I20260924 00:55:14.438153 23134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41531 every 8 connection(s)
I20260924 00:55:14.439476 23068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:14.443179 23135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:14.446794 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 23068
I20260924 00:55:14.446970 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:14.447388 888 external_mini_cluster.cc:1428] Setting key d2aa93aacfb0b7c90f0d457a91920892
I20260924 00:55:14.455968 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:14.449401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41669 (local address 127.0.222.62:41531)
0924 00:55:14.450252 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:14.450267 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:14.450318 (+ 51us) server_negotiation.cc:407] Connection header received
0924 00:55:14.452209 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:14.452237 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:14.452751 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:14.453241 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:14.454350 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:14.454357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:14.454370 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:14.454385 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:14.454446 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:14.454469 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:14.454693 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:14.454820 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 00:55:14.455029 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":347,"threads_started":1}
I20260924 00:55:14.456323 23135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538: Bootstrap starting.
I20260924 00:55:14.459851 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:14.460162 23135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:14.461141 23135 log.cc:824] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:14.464161 23135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538: No bootstrap required, opened a new log
I20260924 00:55:14.466818 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:14.464308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51531 (local address 127.0.222.62:41531)
0924 00:55:14.464457 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:14.464463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:14.464849 (+ 386us) server_negotiation.cc:407] Connection header received
0924 00:55:14.465022 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:14.465027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:14.465079 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:14.465155 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:14.466168 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:14.466174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:14.466179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:14.466197 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:14.466237 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:14.466252 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:14.466358 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:14.466459 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 00:55:14.466526 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:55:14.472082 23135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dfd0eafccd498e83ee47291fb44538" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41531 } }
I20260924 00:55:14.472445 23135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:14.472543 23135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07dfd0eafccd498e83ee47291fb44538, State: Initialized, Role: FOLLOWER
I20260924 00:55:14.473106 23135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dfd0eafccd498e83ee47291fb44538" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41531 } }
I20260924 00:55:14.473369 23135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:14.473496 23135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:14.473670 23135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:14.475935 23135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dfd0eafccd498e83ee47291fb44538" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41531 } }
I20260924 00:55:14.476418 23135 leader_election.cc:302] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07dfd0eafccd498e83ee47291fb44538; no voters:
I20260924 00:55:14.477002 23135 leader_election.cc:288] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:14.477157 23145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:14.477789 23145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [term 1 LEADER]: Becoming Leader. State: Replica: 07dfd0eafccd498e83ee47291fb44538, State: Running, Role: LEADER
I20260924 00:55:14.478454 23145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dfd0eafccd498e83ee47291fb44538" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41531 } }
I20260924 00:55:14.479141 23135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:14.481528 23147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07dfd0eafccd498e83ee47291fb44538. Latest consensus state: current_term: 1 leader_uuid: "07dfd0eafccd498e83ee47291fb44538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dfd0eafccd498e83ee47291fb44538" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41531 } } }
I20260924 00:55:14.481550 23146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07dfd0eafccd498e83ee47291fb44538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07dfd0eafccd498e83ee47291fb44538" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41531 } } }
I20260924 00:55:14.481997 23146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:14.481997 23147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:14.484084 23154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:14.488919 23154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:14.498040 23154 catalog_manager.cc:1347] Generated new cluster ID: dc52db22533a469995128305c8a4d13d
I20260924 00:55:14.498152 23154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:14.528273 23154 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:14.528673 23154 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:14.539777 23154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07dfd0eafccd498e83ee47291fb44538: Generated new TSK 0
I20260924 00:55:14.540653 23154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:14.554114 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23068
W20260924 00:55:14.555655 23130 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41669
W20260924 00:55:14.556454 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23068: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:55:14.556595 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 23068
2026-09-24T00:55:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1805 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:14 dist-test-slave-rtvz krb5kdc[23166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:14 dist-test-slave-rtvz krb5kdc[23166](info): set up 2 sockets
Sep 24 00:55:14 dist-test-slave-rtvz krb5kdc[23166](info): commencing operation
krb5kdc: starting...
W20260924 00:55:15.580648 888 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:15 dist-test-slave-rtvz krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:55:15.833039 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:35257
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:35257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:16.158121 23179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:16.158424 23179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:16.158561 23179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:16.165787 23179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:16.165907 23179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:16.165985 23179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:16.166071 23179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:16.166136 23179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:16.175765 23179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:35257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:16.177023 23179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:16.178313 23179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:16.189780 23184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:16.189741 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:16.189811 23187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:16.190542 23179 server_base.cc:1034] running on GCE node
I20260924 00:55:16.191352 23179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:16.192876 23179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:16.194130 23179 hybrid_clock.cc:648] HybridClock initialized: now 1790211316194034 us; error 41 us; skew 500 ppm
I20260924 00:55:16.194608 23179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:16 dist-test-slave-rtvz krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:16.197921 23179 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:55:16.199645 23179 webserver.cc:460] Webserver started at http://127.0.222.62:46401/ using document root <none> and password file <none>
I20260924 00:55:16.200419 23179 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:16.200548 23179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:16.200858 23179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:16.204010 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "35c246146a8e4ac0b47b914883b7b8b0"
format_stamp: "Formatted at 2026-09-24 00:55:16 on dist-test-slave-rtvz"
server_key: "df383963d667b6a7696a365f4d6f582f"
server_key_iv: "0509ab3dacb9c0128716bd42299aad4b"
server_key_version: "encryptionkey@0"
I20260924 00:55:16.204777 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "35c246146a8e4ac0b47b914883b7b8b0"
format_stamp: "Formatted at 2026-09-24 00:55:16 on dist-test-slave-rtvz"
server_key: "df383963d667b6a7696a365f4d6f582f"
server_key_iv: "0509ab3dacb9c0128716bd42299aad4b"
server_key_version: "encryptionkey@0"
I20260924 00:55:16.210153 23179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:55:16.213840 23194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:16.215325 23179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 00:55:16.215512 23179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "35c246146a8e4ac0b47b914883b7b8b0"
format_stamp: "Formatted at 2026-09-24 00:55:16 on dist-test-slave-rtvz"
server_key: "df383963d667b6a7696a365f4d6f582f"
server_key_iv: "0509ab3dacb9c0128716bd42299aad4b"
server_key_version: "encryptionkey@0"
I20260924 00:55:16.215706 23179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:16.244429 23179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:16.248667 23179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:16.248940 23179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:16.266429 23179 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:35257
I20260924 00:55:16.266474 23245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:35257 every 8 connection(s)
I20260924 00:55:16.267803 23179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:16.271405 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:16.277009 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 23179
I20260924 00:55:16.277235 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:16.277786 888 external_mini_cluster.cc:1428] Setting key f5121349fc4d9c8d43401c7567457205
I20260924 00:55:16.283092 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0: Bootstrap starting.
I20260924 00:55:16.286145 23246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:16.287154 23246 log.cc:824] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:16.290560 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0: No bootstrap required, opened a new log
Sep 24 00:55:16 dist-test-slave-rtvz krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:55:16.300280 23246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c246146a8e4ac0b47b914883b7b8b0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35257 } }
I20260924 00:55:16.300781 23246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:16.300889 23246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c246146a8e4ac0b47b914883b7b8b0, State: Initialized, Role: FOLLOWER
I20260924 00:55:16.301661 23246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c246146a8e4ac0b47b914883b7b8b0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35257 } }
I20260924 00:55:16.301966 23246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:16.302107 23246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:16.302286 23246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:16.302731 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:16.279723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35943 (local address 127.0.222.62:35257)
0924 00:55:16.280611 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:16.280625 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:16.280675 (+ 50us) server_negotiation.cc:407] Connection header received
0924 00:55:16.282605 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:16.282628 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:16.283088 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:16.283499 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:16.285292 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:16.286622 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:16.288139 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:16.289096 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:16.289313 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:16.293412 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:16.293422 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:16.293444 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:16.293490 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:16.297523 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:16.298366 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:16.298375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:16.298385 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:16.298673 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:16.299230 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:16.299235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:16.299238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:16.300017 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:16.300314 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:16.300727 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 00:55:16.301243 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":403,"threads_started":1}
I20260924 00:55:16.304750 23246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c246146a8e4ac0b47b914883b7b8b0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35257 } }
I20260924 00:55:16.305321 23246 leader_election.cc:302] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35c246146a8e4ac0b47b914883b7b8b0; no voters:
I20260924 00:55:16.305961 23246 leader_election.cc:288] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:16.306066 23251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:16.306787 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:16.307904 23246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:16.312206 23251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [term 1 LEADER]: Becoming Leader. State: Replica: 35c246146a8e4ac0b47b914883b7b8b0, State: Running, Role: LEADER
I20260924 00:55:16.312886 23251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c246146a8e4ac0b47b914883b7b8b0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35257 } }
I20260924 00:55:16.315693 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35c246146a8e4ac0b47b914883b7b8b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c246146a8e4ac0b47b914883b7b8b0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35257 } } }
I20260924 00:55:16.316094 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:16.316579 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35c246146a8e4ac0b47b914883b7b8b0. Latest consensus state: current_term: 1 leader_uuid: "35c246146a8e4ac0b47b914883b7b8b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35c246146a8e4ac0b47b914883b7b8b0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 35257 } } }
I20260924 00:55:16.316829 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:16.321707 23266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:16.330884 23266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:16.343933 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:16.312565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43147 (local address 127.0.222.62:35257)
0924 00:55:16.312752 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:16.312758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:16.326239 (+ 13481us) server_negotiation.cc:407] Connection header received
0924 00:55:16.327221 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:16.327228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:16.327328 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:16.327405 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:16.329619 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:16.330685 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:16.332488 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:16.333240 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:16.333421 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:16.336364 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:16.336370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:16.336374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:16.336410 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:16.340218 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:16.341187 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:16.341219 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:16.341224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:16.341394 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:16.341888 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:16.341894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:16.341898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:16.342347 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:16.342582 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:16.343236 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 00:55:16.343497 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:55:16.351521 23266 catalog_manager.cc:1347] Generated new cluster ID: b9d9625919ec4c508b19c61def836b6e
I20260924 00:55:16.351652 23266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:16.372855 23266 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:16.374861 23266 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:16.400396 23266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35c246146a8e4ac0b47b914883b7b8b0: Generated new TSK 0
I20260924 00:55:16.401512 23266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:16.410966 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23179
I20260924 00:55:16.412160 23241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:16.507439 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 23179
2026-09-24T00:55:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:16 dist-test-slave-rtvz krb5kdc[23279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:16 dist-test-slave-rtvz krb5kdc[23279](info): set up 2 sockets
Sep 24 00:55:16 dist-test-slave-rtvz krb5kdc[23279](info): commencing operation
krb5kdc: starting...
W20260924 00:55:17.561509 888 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:17 dist-test-slave-rtvz krb5kdc[23279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:55:17.816231 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33933
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:33933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:18.143702 23292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:18.144001 23292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:18.144098 23292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:18.151345 23292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:18.151480 23292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:18.151538 23292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:18.151628 23292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:18.151681 23292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:18.162084 23292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:33933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:33933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:18.163405 23292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:18.164767 23292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:18.176565 23298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:18.176663 23297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:18.176685 23300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:18.177491 23292 server_base.cc:1034] running on GCE node
I20260924 00:55:18.178304 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:18.179796 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:18.181031 23292 hybrid_clock.cc:648] HybridClock initialized: now 1790211318181002 us; error 73 us; skew 500 ppm
I20260924 00:55:18.181420 23292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:18 dist-test-slave-rtvz krb5kdc[23279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:18.184474 23292 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:55:18.186045 23292 webserver.cc:460] Webserver started at http://127.0.222.62:42855/ using document root <none> and password file <none>
I20260924 00:55:18.186748 23292 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:18.186846 23292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:18.187103 23292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:18.190109 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "9799e270a24545869b8e4e41df199ebb"
format_stamp: "Formatted at 2026-09-24 00:55:18 on dist-test-slave-rtvz"
server_key: "19f278d3fa11673adc26fbcf130e7ba3"
server_key_iv: "ba14d281edd26191eb6dd58295d77d3b"
server_key_version: "encryptionkey@0"
I20260924 00:55:18.190848 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "9799e270a24545869b8e4e41df199ebb"
format_stamp: "Formatted at 2026-09-24 00:55:18 on dist-test-slave-rtvz"
server_key: "19f278d3fa11673adc26fbcf130e7ba3"
server_key_iv: "ba14d281edd26191eb6dd58295d77d3b"
server_key_version: "encryptionkey@0"
I20260924 00:55:18.196206 23292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:55:18.199844 23307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:18.201414 23292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:55:18.201582 23292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "9799e270a24545869b8e4e41df199ebb"
format_stamp: "Formatted at 2026-09-24 00:55:18 on dist-test-slave-rtvz"
server_key: "19f278d3fa11673adc26fbcf130e7ba3"
server_key_iv: "ba14d281edd26191eb6dd58295d77d3b"
server_key_version: "encryptionkey@0"
I20260924 00:55:18.201787 23292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:18.220489 23292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:18.224886 23292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:18.225165 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:18.243810 23292 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:33933
I20260924 00:55:18.243830 23358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:33933 every 8 connection(s)
I20260924 00:55:18.245187 23292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:18.248900 23359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:18.251335 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 23292
I20260924 00:55:18.251518 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:18.252064 888 external_mini_cluster.cc:1428] Setting key 33d852f9d03b4d10f60cd1e539245189
I20260924 00:55:18.261373 23359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb: Bootstrap starting.
I20260924 00:55:18.265035 23359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:18.265951 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:18.253958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53765 (local address 127.0.222.62:33933)
0924 00:55:18.254801 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:18.254813 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:18.254856 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:55:18.256506 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:18.256530 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:18.256936 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:18.257296 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:18.257866 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:18.259285 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:18.261132 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:18.261953 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:18.262094 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:18.263436 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:18.263444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:18.263479 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:18.264045 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:18.264132 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:18.264159 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:18.264264 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:18.264513 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 00:55:18.264797 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":369,"threads_started":1}
I20260924 00:55:18.266188 23359 log.cc:824] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:18.269748 23359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb: No bootstrap required, opened a new log
I20260924 00:55:18.270773 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:18.280519 23359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9799e270a24545869b8e4e41df199ebb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33933 } }
I20260924 00:55:18.281116 23359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:18.281324 23359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9799e270a24545869b8e4e41df199ebb, State: Initialized, Role: FOLLOWER
I20260924 00:55:18.282009 23359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9799e270a24545869b8e4e41df199ebb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33933 } }
I20260924 00:55:18.282248 23359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:18.282400 23359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:18.282588 23359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:18.283200 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:18.276302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41211 (local address 127.0.222.62:33933)
0924 00:55:18.276503 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:18.276509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:18.276967 (+ 458us) server_negotiation.cc:407] Connection header received
0924 00:55:18.277238 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:18.277245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:18.277337 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:18.277475 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:18.277991 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:18.279150 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:18.280529 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:18.281285 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:18.281436 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:18.282472 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:18.282477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:18.282482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:18.282514 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:18.282565 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:18.282590 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:18.282666 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:18.282800 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 00:55:18.282894 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 00:55:18.285701 23359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9799e270a24545869b8e4e41df199ebb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33933 } }
I20260924 00:55:18.286264 23359 leader_election.cc:302] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9799e270a24545869b8e4e41df199ebb; no voters:
I20260924 00:55:18.286845 23359 leader_election.cc:288] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:18.287057 23369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:18.287765 23369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [term 1 LEADER]: Becoming Leader. State: Replica: 9799e270a24545869b8e4e41df199ebb, State: Running, Role: LEADER
I20260924 00:55:18.288286 23369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9799e270a24545869b8e4e41df199ebb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33933 } }
I20260924 00:55:18.289084 23359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:18.291029 23371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9799e270a24545869b8e4e41df199ebb. Latest consensus state: current_term: 1 leader_uuid: "9799e270a24545869b8e4e41df199ebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9799e270a24545869b8e4e41df199ebb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33933 } } }
I20260924 00:55:18.291476 23371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:18.292363 23370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9799e270a24545869b8e4e41df199ebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9799e270a24545869b8e4e41df199ebb" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 33933 } } }
I20260924 00:55:18.292722 23370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:18.294059 23374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:18.298966 23374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:18.307739 23374 catalog_manager.cc:1347] Generated new cluster ID: 5c0b5c7f26e14485813e2f5f244d0060
I20260924 00:55:18.307858 23374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:18.328472 23374 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:18.330523 23374 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:18.351138 23374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9799e270a24545869b8e4e41df199ebb: Generated new TSK 0
I20260924 00:55:18.352340 23374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:18.369411 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23292
W20260924 00:55:18.370721 23354 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53765
W20260924 00:55:18.371567 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23292: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:55:18.371745 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 23292
2026-09-24T00:55:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1863 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:18 dist-test-slave-rtvz krb5kdc[23390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:18 dist-test-slave-rtvz krb5kdc[23390](info): set up 2 sockets
Sep 24 00:55:18 dist-test-slave-rtvz krb5kdc[23390](info): commencing operation
krb5kdc: starting...
W20260924 00:55:19.404786 888 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:19 dist-test-slave-rtvz krb5kdc[23390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:55:19.662827 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34025
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:34025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:19.994699 23403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:19.995019 23403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:19.995158 23403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:20.002626 23403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:20.002758 23403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:20.002830 23403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:20.002914 23403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:20.002972 23403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:20.013166 23403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:34025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:34025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:20.014539 23403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:20.015895 23403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:20.027347 23411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:20.027281 23408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:20.027786 23409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:20.028282 23403 server_base.cc:1034] running on GCE node
I20260924 00:55:20.029039 23403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:20.030578 23403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:20.031800 23403 hybrid_clock.cc:648] HybridClock initialized: now 1790211320031743 us; error 48 us; skew 500 ppm
I20260924 00:55:20.032145 23403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:20 dist-test-slave-rtvz krb5kdc[23390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:20.040930 23403 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:55:20.042636 23403 webserver.cc:460] Webserver started at http://127.0.222.62:39947/ using document root <none> and password file <none>
I20260924 00:55:20.043375 23403 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:20.043485 23403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:20.043771 23403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:20.046958 23403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "f96a9549160b47acb644697d803c79e0"
format_stamp: "Formatted at 2026-09-24 00:55:20 on dist-test-slave-rtvz"
server_key: "d5a9a83afb87ff6888e5528012106189"
server_key_iv: "0c1115ff46bd4587ad444e605c6d03cf"
server_key_version: "encryptionkey@0"
I20260924 00:55:20.047731 23403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "f96a9549160b47acb644697d803c79e0"
format_stamp: "Formatted at 2026-09-24 00:55:20 on dist-test-slave-rtvz"
server_key: "d5a9a83afb87ff6888e5528012106189"
server_key_iv: "0c1115ff46bd4587ad444e605c6d03cf"
server_key_version: "encryptionkey@0"
I20260924 00:55:20.053149 23403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 00:55:20.056977 23418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:20.058638 23403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:55:20.058797 23403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "f96a9549160b47acb644697d803c79e0"
format_stamp: "Formatted at 2026-09-24 00:55:20 on dist-test-slave-rtvz"
server_key: "d5a9a83afb87ff6888e5528012106189"
server_key_iv: "0c1115ff46bd4587ad444e605c6d03cf"
server_key_version: "encryptionkey@0"
I20260924 00:55:20.059003 23403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:20.064785 23403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:20.068944 23403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:20.069257 23403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:20.087059 23403 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:34025
I20260924 00:55:20.087157 23469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:34025 every 8 connection(s)
I20260924 00:55:20.088662 23403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:20.092669 23470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:20.098430 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 23403
I20260924 00:55:20.098620 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:20.099160 888 external_mini_cluster.cc:1428] Setting key ff838210d1add542a2cf78aa383a4ba3
I20260924 00:55:20.104341 23470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0: Bootstrap starting.
I20260924 00:55:20.107409 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:20.101120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49313 (local address 127.0.222.62:34025)
0924 00:55:20.102092 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:20.102104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:20.102156 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:55:20.103947 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:20.103977 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:20.104357 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:20.104726 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:20.105776 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:20.105782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:20.105796 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:20.105815 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:20.105881 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:20.105905 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:20.106146 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:20.106299 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:55:20.106479 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":428,"threads_started":1}
I20260924 00:55:20.107914 23470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:20.108973 23470 log.cc:824] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:20.111147 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:20.111907 23470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0: No bootstrap required, opened a new log
I20260924 00:55:20.117863 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:20.115405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42885 (local address 127.0.222.62:34025)
0924 00:55:20.115566 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:20.115572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:20.115979 (+ 407us) server_negotiation.cc:407] Connection header received
0924 00:55:20.116152 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:20.116156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:20.116213 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:20.116307 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:20.117171 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:20.117176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:20.117181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:55:20.117231 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:20.117278 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:55:20.117297 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:20.117410 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:20.117521 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 00:55:20.117586 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:55:20.120185 23470 raft_consensus.cc:348] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a9549160b47acb644697d803c79e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34025 } }
I20260924 00:55:20.120579 23470 raft_consensus.cc:374] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:55:20.120668 23470 raft_consensus.cc:729] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96a9549160b47acb644697d803c79e0, State: Initialized, Role: FOLLOWER
I20260924 00:55:20.121268 23470 consensus_queue.cc:260] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a9549160b47acb644697d803c79e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34025 } }
I20260924 00:55:20.121531 23470 raft_consensus.cc:388] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:55:20.121659 23470 raft_consensus.cc:482] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:55:20.121816 23470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:20.124311 23470 raft_consensus.cc:504] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a9549160b47acb644697d803c79e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34025 } }
I20260924 00:55:20.124806 23470 leader_election.cc:302] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f96a9549160b47acb644697d803c79e0; no voters:
I20260924 00:55:20.125411 23470 leader_election.cc:288] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:55:20.125519 23480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:20.126137 23480 raft_consensus.cc:686] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [term 1 LEADER]: Becoming Leader. State: Replica: f96a9549160b47acb644697d803c79e0, State: Running, Role: LEADER
I20260924 00:55:20.126730 23480 consensus_queue.cc:237] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a9549160b47acb644697d803c79e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34025 } }
I20260924 00:55:20.127516 23470 sys_catalog.cc:564] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:20.129669 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f96a9549160b47acb644697d803c79e0. Latest consensus state: current_term: 1 leader_uuid: "f96a9549160b47acb644697d803c79e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a9549160b47acb644697d803c79e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34025 } } }
I20260924 00:55:20.130151 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:20.130712 23481 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f96a9549160b47acb644697d803c79e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96a9549160b47acb644697d803c79e0" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 34025 } } }
I20260924 00:55:20.131121 23481 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:20.131579 23489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:20.137358 23489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:20.148006 23489 catalog_manager.cc:1347] Generated new cluster ID: fa083de21c0b47f7a3c83265024e7346
I20260924 00:55:20.148134 23489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:20.157948 23489 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:20.158443 23489 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:20.178687 23489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f96a9549160b47acb644697d803c79e0: Generated new TSK 0
I20260924 00:55:20.179847 23489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:20.187433 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23403
W20260924 00:55:20.188591 23465 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49313
W20260924 00:55:20.189282 888 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23403: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 00:55:20.189411 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 23403
2026-09-24T00:55:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1818 ms)
[----------] 18 tests from AuthTokenIssuingTest (33491 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:20 dist-test-slave-rtvz krb5kdc[23501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:20 dist-test-slave-rtvz krb5kdc[23501](info): set up 2 sockets
Sep 24 00:55:20 dist-test-slave-rtvz krb5kdc[23501](info): commencing operation
krb5kdc: starting...
W20260924 00:55:21.212790 888 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:21 dist-test-slave-rtvz krb5kdc[23501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:55:21.466598 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41621
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41621,127.0.222.61:44357,127.0.222.60:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:21.799580 23514 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:55:21.799904 23514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:21.800004 23514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:21.800122 23514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:21.807312 23514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:21.807415 23514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:21.807474 23514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:21.807542 23514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:21.807626 23514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:21.817629 23514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41621,127.0.222.61:44357,127.0.222.60:32953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:21.818965 23514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:21.820266 23514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:21.831660 23519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:21.831745 23522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:21.831772 23520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:21.832651 23514 server_base.cc:1034] running on GCE node
I20260924 00:55:21.833457 23514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:21.834836 23514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:21.836115 23514 hybrid_clock.cc:648] HybridClock initialized: now 1790211321835967 us; error 59 us; skew 500 ppm
I20260924 00:55:21.836443 23514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:21 dist-test-slave-rtvz krb5kdc[23501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:21.839519 23514 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:55:21.840931 23514 webserver.cc:460] Webserver started at http://127.0.222.62:43423/ using document root <none> and password file <none>
I20260924 00:55:21.841717 23514 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:21.841812 23514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:21.842127 23514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:21.845086 23514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "41d23ad8fe91473a89e304cfc6511222"
format_stamp: "Formatted at 2026-09-24 00:55:21 on dist-test-slave-rtvz"
server_key: "9ce1ad32730fc7fef4fb1a6c1ecbe2ab"
server_key_iv: "d357fefc846ea6ebaaeb89effa6e88e1"
server_key_version: "encryptionkey@0"
I20260924 00:55:21.845920 23514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "41d23ad8fe91473a89e304cfc6511222"
format_stamp: "Formatted at 2026-09-24 00:55:21 on dist-test-slave-rtvz"
server_key: "9ce1ad32730fc7fef4fb1a6c1ecbe2ab"
server_key_iv: "d357fefc846ea6ebaaeb89effa6e88e1"
server_key_version: "encryptionkey@0"
I20260924 00:55:21.851213 23514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:55:21.854823 23529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:21.856343 23514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 00:55:21.856513 23514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "41d23ad8fe91473a89e304cfc6511222"
format_stamp: "Formatted at 2026-09-24 00:55:21 on dist-test-slave-rtvz"
server_key: "9ce1ad32730fc7fef4fb1a6c1ecbe2ab"
server_key_iv: "d357fefc846ea6ebaaeb89effa6e88e1"
server_key_version: "encryptionkey@0"
I20260924 00:55:21.856731 23514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:21.876704 23514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:21.881809 23514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:21.882073 23514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:21.899446 23514 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:41621
I20260924 00:55:21.899464 23580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:41621 every 8 connection(s)
I20260924 00:55:21.900843 23514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:21.901352 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 23514
I20260924 00:55:21.901592 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:21.902328 888 external_mini_cluster.cc:1428] Setting key b6cb87185925edd4ded1304634e1c881
I20260924 00:55:21.904932 23581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:21.912542 23581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.222.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.61@KRBTEST.COM" created.
Sep 24 00:55:21 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:55:21.939754 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:21.917189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41621 (local address 127.0.0.1:36432)
0924 00:55:21.918099 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:21.918151 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:21.920528 (+ 2377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:21.921727 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:21.921733 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:21.922520 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:21.922933 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:21.922954 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:21.925328 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:21.925333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:21.926577 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:21.926590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:21.927975 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:21.927981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:21.928132 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:21.928792 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:21.928818 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:21.931338 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:21.935186 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:21.935198 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:21.935208 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:21.935806 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:21.936323 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:21.936326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:21.936329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:21.936529 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:21.937557 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:21.937563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:21.937780 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 00:55:21.938009 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 00:55:21.938702 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":203,"thread_start_us":362,"threads_started":1}
I20260924 00:55:21.939812 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:21.918630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.0.222.62:41621)
0924 00:55:21.919375 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:21.919384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:21.919444 (+ 60us) server_negotiation.cc:407] Connection header received
0924 00:55:21.921007 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:21.921016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:21.921412 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:21.921681 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:21.923174 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:21.925141 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:21.926752 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:21.927716 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:21.927882 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:21.931609 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:21.931622 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:21.931641 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:21.931679 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:21.935011 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:21.935957 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:21.935966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:21.935969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:21.936172 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:21.936641 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:21.936645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:21.936648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:21.937391 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:21.937581 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:21.938178 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 00:55:21.938527 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":347,"threads_started":1}
I20260924 00:55:21.943346 23581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:55:21.944517 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:21.944108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41086)
0924 00:55:21.944281 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:21.944395 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 00:55:21.944895 23530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
I20260924 00:55:21.946487 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:21.946130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41100)
0924 00:55:21.946271 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:21.946347 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 00:55:21.946688 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:44357: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab.
I20260924 00:55:21.965282 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:44357
--webserver_interface=127.0.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41621,127.0.222.61:44357,127.0.222.60:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:55:21.996045 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } attempt: 1
I20260924 00:55:21.997416 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:21.996854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41104)
0924 00:55:21.997070 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:21.997279 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 00:55:21.998531 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:21.998122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41108)
0924 00:55:21.998329 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:21.998405 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 00:55:21.998733 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:44357: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
I20260924 00:55:22.066987 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } attempt: 2
I20260924 00:55:22.068322 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.067759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41114)
0924 00:55:22.068047 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.068158 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 00:55:22.069465 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.069063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41116)
0924 00:55:22.069244 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.069333 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 00:55:22.069676 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:44357: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
I20260924 00:55:22.160952 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } attempt: 3
I20260924 00:55:22.162129 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.161693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41128)
0924 00:55:22.161890 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.161990 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 00:55:22.163223 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.162862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41132)
0924 00:55:22.163002 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.163085 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 00:55:22.163430 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:44357: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
W20260924 00:55:22.298341 23588 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:55:22.298692 23588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:22.298766 23588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:22.298869 23588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:22.306246 23588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:22.306348 23588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:22.306403 23588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:22.306453 23588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:22.306519 23588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:22.306677 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } attempt: 4
I20260924 00:55:22.307960 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.307374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41144)
0924 00:55:22.307722 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.307818 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260924 00:55:22.308995 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.308642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41160)
0924 00:55:22.308782 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.308870 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 00:55:22.309265 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:44357: Network error: Client connection negotiation failed: client connection to 127.0.222.61:44357: connect: Connection refused (error 111)
I20260924 00:55:22.317423 23588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41621,127.0.222.61:44357,127.0.222.60:32953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:44357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:22.318778 23588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:22.320178 23588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:22.331964 23596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:22.331928 23593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:22.332003 23594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:22.332899 23588 server_base.cc:1034] running on GCE node
I20260924 00:55:22.333931 23588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:22.335546 23588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:22.336790 23588 hybrid_clock.cc:648] HybridClock initialized: now 1790211322336707 us; error 60 us; skew 500 ppm
I20260924 00:55:22.337150 23588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:22.340440 23588 init.cc:377] Logged in from keytab as kudu/127.0.222.61@KRBTEST.COM (short username kudu)
I20260924 00:55:22.342080 23588 webserver.cc:460] Webserver started at http://127.0.222.61:32945/ using document root <none> and password file <none>
I20260924 00:55:22.342777 23588 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:22.342881 23588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:22.343200 23588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:22.346349 23588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/data/instance:
uuid: "9670171074484c6ea78d9efde06c1313"
format_stamp: "Formatted at 2026-09-24 00:55:22 on dist-test-slave-rtvz"
server_key: "8521bad0b55799b759948786bef0beb3"
server_key_iv: "4396b000b6bfc9c76b29d75761c742f1"
server_key_version: "encryptionkey@0"
I20260924 00:55:22.347147 23588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/wal/instance:
uuid: "9670171074484c6ea78d9efde06c1313"
format_stamp: "Formatted at 2026-09-24 00:55:22 on dist-test-slave-rtvz"
server_key: "8521bad0b55799b759948786bef0beb3"
server_key_iv: "4396b000b6bfc9c76b29d75761c742f1"
server_key_version: "encryptionkey@0"
I20260924 00:55:22.352725 23588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 00:55:22.356375 23603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:22.357947 23588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 00:55:22.358135 23588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/wal
uuid: "9670171074484c6ea78d9efde06c1313"
format_stamp: "Formatted at 2026-09-24 00:55:22 on dist-test-slave-rtvz"
server_key: "8521bad0b55799b759948786bef0beb3"
server_key_iv: "4396b000b6bfc9c76b29d75761c742f1"
server_key_version: "encryptionkey@0"
I20260924 00:55:22.358349 23588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:22.373140 23588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:22.377583 23588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:22.377836 23588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:22.396639 23588 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:44357
I20260924 00:55:22.396678 23654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:44357 every 8 connection(s)
I20260924 00:55:22.398156 23588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
I20260924 00:55:22.400966 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 23588
I20260924 00:55:22.401160 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-1/wal/instance
I20260924 00:55:22.401746 888 external_mini_cluster.cc:1428] Setting key af0b90fa9f7db39d73beadac94da9499
I20260924 00:55:22.402563 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:22.411351 23655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.60@KRBTEST.COM" created.
I20260924 00:55:22.434587 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.414761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.0.222.62:41621)
0924 00:55:22.414984 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.414991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.416266 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 00:55:22.418349 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.418356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.418460 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.418578 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.420248 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.421237 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.423102 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.423728 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.423864 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.427946 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.427954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.427959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.428002 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.430964 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.432052 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.432058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.432061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.432220 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.432727 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.432730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.432732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.433016 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.433141 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.434109 (+ 968us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.434317 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 00:55:22.435902 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.415163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41621 (local address 127.0.0.1:36442)
0924 00:55:22.416059 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.416108 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:22.418051 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:22.418808 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:22.418817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:22.419413 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:22.420053 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.420074 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.421447 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.421452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.422893 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.422911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.423987 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.423996 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.424297 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.425110 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:22.425144 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:22.427612 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:22.431158 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.431169 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.431196 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.431903 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.432343 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.432349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.432353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.432627 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.433141 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:22.433150 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:22.433709 (+ 559us) client_negotiation.cc:769] Sending connection context
0924 00:55:22.434012 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 00:55:22.434543 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":369,"threads_started":1}
I20260924 00:55:22.437765 23655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:55:22.452505 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.438560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41166)
0924 00:55:22.438740 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.438759 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:22.438865 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:22.439830 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:22.439834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:22.440216 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:22.440372 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.440380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.441878 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.441881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.442758 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.442770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.443800 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.443805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.443928 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.444460 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:22.444476 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:22.446606 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:22.450212 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.450216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.450219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.450553 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.450909 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.450912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.450915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.451041 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.451723 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:22.451726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:22.451852 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 00:55:22.451966 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 00:55:22.452242 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 00:55:22.453012 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.438685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.0.222.61:44357)
0924 00:55:22.439333 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.439340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.439371 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:55:22.439422 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.439429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.439676 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.439821 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.440536 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.441743 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.442886 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.443663 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.443771 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.446816 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.446830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.446850 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.446885 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.450082 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.450680 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.450688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.450692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.450822 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.451156 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.451160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.451163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.451599 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.451788 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.452166 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.452439 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":284,"threads_started":1}
I20260924 00:55:22.455598 23655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:55:22.456723 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.456173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58108)
0924 00:55:22.456381 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.456536 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 00:55:22.456976 23604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
I20260924 00:55:22.458710 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.458275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58110)
0924 00:55:22.458493 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.458591 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 00:55:22.458922 23655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab.
I20260924 00:55:22.468447 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:32953
--webserver_interface=127.0.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:41621,127.0.222.61:44357,127.0.222.60:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:55:22.508275 23655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 1
I20260924 00:55:22.509681 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.509142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58116)
0924 00:55:22.509409 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.509518 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 00:55:22.510870 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.510494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58120)
0924 00:55:22.510668 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.510747 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 00:55:22.511093 23655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
I20260924 00:55:22.579329 23655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 2
I20260924 00:55:22.580564 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.580079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58136)
0924 00:55:22.580300 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.580398 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 00:55:22.581655 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.581292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58150)
0924 00:55:22.581440 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.581519 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 00:55:22.581868 23655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
I20260924 00:55:22.608538 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } attempt: 5
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:55:22.624569 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.609490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41174)
0924 00:55:22.609728 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.609751 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:22.609884 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:22.610245 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:22.610250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:22.610758 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:22.610985 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.610999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.612152 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.612157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.613568 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.613587 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.614452 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.614459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.614700 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.615400 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:22.615425 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:22.617955 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:22.622035 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.622039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.622042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.622490 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.622911 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.622915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.622918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.623079 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.623719 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:22.623722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:22.623884 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 00:55:22.624023 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 00:55:22.624265 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 00:55:22.624692 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.609737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.0.222.61:44357)
0924 00:55:22.609897 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.609902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.609922 (+ 20us) server_negotiation.cc:407] Connection header received
0924 00:55:22.610004 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.610008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.610095 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.610179 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.611202 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.611992 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.613754 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.614255 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.614369 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.618219 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.618227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.618232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.618278 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.621831 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.622625 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.622630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.622634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.622792 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.623186 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.623191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.623194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.623574 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.623727 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.624147 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.624387 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 00:55:22.625730 23581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:55:22.626765 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.626294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58166)
0924 00:55:22.626484 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.626587 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 00:55:22.627820 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.627473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58174)
0924 00:55:22.627635 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.627718 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 00:55:22.628026 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
I20260924 00:55:22.673132 23655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 3
I20260924 00:55:22.674456 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.673922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58186)
0924 00:55:22.674194 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.674296 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 00:55:22.675674 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.675237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58196)
0924 00:55:22.675464 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.675548 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260924 00:55:22.675873 23655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
I20260924 00:55:22.679250 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 1
I20260924 00:55:22.680377 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.679872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58208)
0924 00:55:22.680058 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.680161 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 00:55:22.681352 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.680984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58218)
0924 00:55:22.681130 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.681234 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 00:55:22.681548 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
I20260924 00:55:22.749778 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 2
I20260924 00:55:22.751005 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.750510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58220)
0924 00:55:22.750743 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.750836 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 00:55:22.752079 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.751717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58226)
0924 00:55:22.751877 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.751961 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 00:55:22.752293 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
W20260924 00:55:22.812995 23662 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:55:22.813387 23662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:22.813485 23662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:22.813601 23662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 00:55:22.819114 23655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 4
I20260924 00:55:22.820315 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.819769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58236)
0924 00:55:22.820004 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.820152 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 00:55:22.821452 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.821060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58252)
0924 00:55:22.821233 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.821320 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 00:55:22.821761 23662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:22.821871 23662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:22.821969 23662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:22.822042 23662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:22.822144 23662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:55:22.821808 23655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
I20260924 00:55:22.834731 23662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:41621,127.0.222.61:44357,127.0.222.60:32953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:32953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:22.836012 23662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:22.837402 23662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:22.849134 23668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:22.849231 23670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:22.849182 23667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:22.850419 23662 server_base.cc:1034] running on GCE node
I20260924 00:55:22.851265 23662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:22.852774 23662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:22.854015 23662 hybrid_clock.cc:648] HybridClock initialized: now 1790211322853926 us; error 52 us; skew 500 ppm
I20260924 00:55:22.854419 23662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:22.857715 23662 init.cc:377] Logged in from keytab as kudu/127.0.222.60@KRBTEST.COM (short username kudu)
I20260924 00:55:22.858496 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 3
I20260924 00:55:22.859580 23662 webserver.cc:460] Webserver started at http://127.0.222.60:36627/ using document root <none> and password file <none>
I20260924 00:55:22.860270 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.859793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58264)
0924 00:55:22.859999 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.860085 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 00:55:22.860431 23662 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:22.860553 23662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:22.860944 23662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:22.861405 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.860990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58266)
0924 00:55:22.861148 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.861283 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 00:55:22.861600 23581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:32953: Network error: Client connection negotiation failed: client connection to 127.0.222.60:32953: connect: Connection refused (error 111)
I20260924 00:55:22.864459 23662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/data/instance:
uuid: "06402b960c024908afa56558368d930f"
format_stamp: "Formatted at 2026-09-24 00:55:22 on dist-test-slave-rtvz"
server_key: "ae3000cd72480df81af70a3d3f344411"
server_key_iv: "3d0f3f8a74d92b3cb69ba2ad944e2c9d"
server_key_version: "encryptionkey@0"
I20260924 00:55:22.865329 23662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/wal/instance:
uuid: "06402b960c024908afa56558368d930f"
format_stamp: "Formatted at 2026-09-24 00:55:22 on dist-test-slave-rtvz"
server_key: "ae3000cd72480df81af70a3d3f344411"
server_key_iv: "3d0f3f8a74d92b3cb69ba2ad944e2c9d"
server_key_version: "encryptionkey@0"
I20260924 00:55:22.870757 23662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 00:55:22.874423 23677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:22.875954 23662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:55:22.876189 23662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/wal
uuid: "06402b960c024908afa56558368d930f"
format_stamp: "Formatted at 2026-09-24 00:55:22 on dist-test-slave-rtvz"
server_key: "ae3000cd72480df81af70a3d3f344411"
server_key_iv: "3d0f3f8a74d92b3cb69ba2ad944e2c9d"
server_key_version: "encryptionkey@0"
I20260924 00:55:22.876395 23662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:22.898415 23662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:22.903105 23662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:22.903417 23662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:22.921191 23662 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:32953
I20260924 00:55:22.921221 23728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:32953 every 8 connection(s)
I20260924 00:55:22.922626 23662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
I20260924 00:55:22.923804 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 23662
I20260924 00:55:22.923997 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790210962623352-888-0/minicluster-data/master-2/wal/instance
I20260924 00:55:22.924564 888 external_mini_cluster.cc:1428] Setting key 841a2ae7586227d230dd2017151e6e3b
I20260924 00:55:22.927546 23729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:22.937001 23729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:55:22.951241 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.926614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.222.60:32953)
0924 00:55:22.927404 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.927420 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.927471 (+ 51us) server_negotiation.cc:407] Connection header received
0924 00:55:22.929763 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.929794 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.930308 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.930798 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.932283 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.934418 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.935980 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.937158 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.937412 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.942292 (+ 4880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.942303 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.942317 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.942364 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.946302 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.947142 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.947149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.947155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.947342 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.947829 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.947834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.947837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.948561 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.948900 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.949404 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.949895 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":328,"threads_started":1}
I20260924 00:55:22.957307 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.940260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:41621 (local address 127.0.0.1:36452)
0924 00:55:22.941378 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.941430 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:22.941595 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:22.942031 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:22.942045 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:22.943200 (+ 1155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:22.943634 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.943648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.945108 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.945114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.946711 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.946722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.947582 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.947588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.947733 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.948428 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:22.948451 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:22.951109 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:22.954563 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.954576 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.954582 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.955096 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.955484 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.955489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.955493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.955632 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.956182 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:22.956190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:22.956345 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 00:55:22.956494 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 00:55:22.956837 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":660,"threads_started":1}
I20260924 00:55:22.957321 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.940345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.0.222.62:41621)
0924 00:55:22.940970 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.940977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.941531 (+ 554us) server_negotiation.cc:407] Connection header received
0924 00:55:22.941733 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.941739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.941829 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.941944 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.943825 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.944869 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.946878 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.947396 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.947522 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.951371 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.951379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.951384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.951434 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.954393 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.955229 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.955233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.955236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.955374 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.955744 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.955748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.955750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.956022 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.956156 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.956655 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.956882 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":314,"threads_started":1}
I20260924 00:55:22.958871 23729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:55:22.971149 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.954462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.0.222.62:41621)
0924 00:55:22.955152 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.955156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.955175 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:55:22.955217 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.955220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.955273 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.955367 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.956634 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.957662 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.959295 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.959808 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.959922 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.964902 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.964910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.964915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.964963 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.967987 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.968801 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.968806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.968810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.968955 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.969478 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.969482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.969484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.969867 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.970139 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.970492 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.970735 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":295,"threads_started":1}
I20260924 00:55:22.974707 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.959699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:44357 (local address 127.0.0.1:41180)
0924 00:55:22.960024 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.960066 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:22.960169 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:22.960500 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:22.960506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:22.960987 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:22.961269 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.961282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.962387 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.962392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.963700 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.963716 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.964811 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.964817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.965009 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.965601 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:22.965627 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:22.968357 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:22.972098 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.972104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.972108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.972575 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.972995 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.973000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.973004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.973153 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.973712 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:22.973717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:22.973901 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 00:55:22.974046 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 00:55:22.974349 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260924 00:55:22.974823 23659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.959716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.0.222.61:44357)
0924 00:55:22.959891 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.959897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.960134 (+ 237us) server_negotiation.cc:407] Connection header received
0924 00:55:22.960276 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.960282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.960348 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.960433 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.961446 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.962241 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.963873 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.964630 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.964767 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.968564 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.968570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.968573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.968612 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.971945 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.972728 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.972732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.972735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.972889 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.973289 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.973295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.973298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.973564 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.973732 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.974225 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.974416 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:55:22.975821 23729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
Sep 24 00:55:22 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260924 00:55:22.990640 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.976816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.0.222.60:32953)
0924 00:55:22.976979 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.976985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.977003 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:55:22.977049 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.977054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.977123 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.977264 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.978294 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.979262 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.981190 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.981759 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.981877 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.985341 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.985346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.985350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.985382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.987979 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.988641 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.988645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.988649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.988780 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.989145 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.989148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.989150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.989503 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.989635 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.989971 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.990212 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
I20260924 00:55:22.990734 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.976580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58282)
0924 00:55:22.976769 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:22.976787 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:22.976890 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:22.977298 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:22.977303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:22.977832 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:22.978116 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.978130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.979416 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.979421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.980762 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:22.980780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.981950 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.981957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:22.982188 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.982777 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:22.982805 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:22.985069 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:22.988150 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.988156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.988160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.988524 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.988901 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:22.988905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:22.988909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:22.989045 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:22.989611 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:22.989615 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:22.989749 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 00:55:22.989876 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 00:55:22.990384 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":169}
I20260924 00:55:22.990792 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:22.973039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.0.222.61:44357)
0924 00:55:22.974164 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:22.974170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:22.974192 (+ 22us) server_negotiation.cc:407] Connection header received
0924 00:55:22.974238 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:22.974242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:22.974313 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:22.974410 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:22.975964 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.977120 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.978871 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:22.979492 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:22.979706 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:22.985096 (+ 5390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:22.985103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:22.985107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:22.985147 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:22.987914 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.988681 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.988686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.988688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.988824 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:22.989254 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:22.989258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:22.989260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:22.989529 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:22.989786 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:22.990112 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 00:55:22.990367 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973,"thread_start_us":801,"threads_started":1}
I20260924 00:55:22.996474 23729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: Bootstrap starting.
I20260924 00:55:22.996704 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:23.000068 23729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:23.001242 23729 log.cc:824] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:22.999610 23556 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "41d23ad8fe91473a89e304cfc6511222"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36460
I20260924 00:55:23.004099 23729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: No bootstrap required, opened a new log
I20260924 00:55:23.011409 23729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } }
I20260924 00:55:23.011801 23729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06402b960c024908afa56558368d930f, State: Initialized, Role: FOLLOWER
I20260924 00:55:23.012252 23729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } }
I20260924 00:55:23.013154 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.013502 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:23.014582 23729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [sys.catalog]: configured and running, proceeding with master startup.
W20260924 00:55:23.024633 23749 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:55:23.024832 23749 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:55:23.022559 23662 master_runner.cc:186] Error getting master registration for 127.0.222.62:41621: OK, instance_id { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" instance_seqno: 1790211321894419 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:55:23.034402 23662 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:55:23.038838 23581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 4
Sep 24 00:55:23 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260924 00:55:23.056629 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.039531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58298)
0924 00:55:23.039743 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:23.039767 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:23.039905 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:23.040335 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:23.040341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:23.040893 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:23.041128 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:23.041141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.042610 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.042616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:23.044109 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:23.044127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.045276 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.045284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:23.045521 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.046238 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:23.046269 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:23.049229 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:23.053731 (+ 4502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:23.053738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:23.053743 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:23.054169 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:23.054638 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:23.054644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:23.054647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:23.054803 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:23.055572 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:23.055577 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:23.055753 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 00:55:23.055879 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 00:55:23.056222 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 00:55:23.056636 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.039631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.0.222.60:32953)
0924 00:55:23.039807 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.039813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.039835 (+ 22us) server_negotiation.cc:407] Connection header received
0924 00:55:23.040063 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.040073 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.040160 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.040285 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.041298 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.042464 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.044292 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.045046 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.045174 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.049484 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:23.049490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:23.049493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:23.049534 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:23.053505 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.054320 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.054330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.054334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.054505 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.054921 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.054928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.054931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.055440 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:23.055621 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.056016 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.056255 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:55:23.062434 23581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222: Bootstrap starting.
I20260924 00:55:23.065904 23581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:23.066965 23581 log.cc:824] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:23.070005 23581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222: No bootstrap required, opened a new log
I20260924 00:55:23.075282 23581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } }
I20260924 00:55:23.075660 23581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d23ad8fe91473a89e304cfc6511222, State: Initialized, Role: FOLLOWER
I20260924 00:55:23.076058 23581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } }
I20260924 00:55:23.076910 23752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.077509 23752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:23.078258 23581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:23.078454 23556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:55:23.078730 23556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:23.082933 23556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } }
I20260924 00:55:23.086287 23704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41d23ad8fe91473a89e304cfc6511222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "06402b960c024908afa56558368d930f"
I20260924 00:55:23.086855 23704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:23.090619 23704 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d23ad8fe91473a89e304cfc6511222 in term 1.
I20260924 00:55:23.091483 23530 leader_election.cc:302] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06402b960c024908afa56558368d930f, 41d23ad8fe91473a89e304cfc6511222; no voters:
I20260924 00:55:23.092160 23752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:23.098354 23556 leader_election.cc:288] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [CANDIDATE]: Term 1 election: Requested vote from peers 9670171074484c6ea78d9efde06c1313 (127.0.222.61:44357), 06402b960c024908afa56558368d930f (127.0.222.60:32953)
I20260924 00:55:23.090340 23514 master_runner.cc:186] Error getting master registration for 127.0.222.61:44357: OK, instance_id { permanent_uuid: "9670171074484c6ea78d9efde06c1313" instance_seqno: 1790211322391155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:55:23.104151 23514 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:55:23.115165 23752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [term 1 LEADER]: Becoming Leader. State: Replica: 41d23ad8fe91473a89e304cfc6511222, State: Running, Role: LEADER
I20260924 00:55:23.116405 23752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } }
I20260924 00:55:23.121994 23655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } attempt: 5
I20260924 00:55:23.126274 23765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d23ad8fe91473a89e304cfc6511222. Latest consensus state: current_term: 1 leader_uuid: "41d23ad8fe91473a89e304cfc6511222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.126971 23765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: This master's current role is: LEADER
Sep 24 00:55:23 dist-test-slave-rtvz krb5kdc[23501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
W20260924 00:55:23.134902 23535 tablet.cc:2369] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:55:23.140338 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.122711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:32953 (local address 127.0.0.1:58302)
0924 00:55:23.123121 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:23.123142 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:23.123280 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:23.124207 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:23.124212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:23.124722 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:23.124942 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:23.124957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.126565 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.126570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:23.128013 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:23.128042 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.129350 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.129357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:23.129599 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.130356 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:23.130380 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:23.133325 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:23.137301 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:23.137307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:23.137311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:23.137832 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:23.138452 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:23.138458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:23.138463 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:23.138646 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:23.139342 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:23.139349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:23.139515 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 00:55:23.139662 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 00:55:23.139951 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260924 00:55:23.141422 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.123506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.0.222.60:32953)
0924 00:55:23.123699 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.123705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.123724 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:55:23.123788 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.123794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.123881 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.123981 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.125342 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.126396 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.128247 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.128966 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.129100 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.133521 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:23.133527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:23.133532 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:23.133573 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:23.137090 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.137973 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.137980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.137984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.138306 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.138779 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.138784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.138788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.139139 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:23.139278 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.140736 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.141006 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 00:55:23.141690 23763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:55:23.141824 23763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:55:23.142520 23774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:23.144728 23774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:23.147272 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313: Bootstrap starting.
I20260924 00:55:23.151501 23655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:23.153240 23655 log.cc:824] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313: Log is configured to *not* fsync() on all Append() calls
W20260924 00:55:23.155720 23683 tablet.cc:2369] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:55:23.162421 23704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [term 1 FOLLOWER]: Refusing update from remote peer 41d23ad8fe91473a89e304cfc6511222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:55:23.164377 23765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:55:23.167654 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313: No bootstrap required, opened a new log
I20260924 00:55:23.179839 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.126741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.0.222.60:32953)
0924 00:55:23.151857 (+ 25116us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.151865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.162415 (+ 10550us) server_negotiation.cc:407] Connection header received
0924 00:55:23.162616 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.162623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.162711 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.162814 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.164523 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.165624 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.167512 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.168219 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.168330 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.171791 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:23.171799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:23.171804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:23.171851 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:23.175721 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.176785 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.176791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.176795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.176954 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.177480 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.177486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.177490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.177970 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:23.178251 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.178915 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.179334 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24948,"thread_start_us":6690,"threads_started":1}
I20260924 00:55:23.186780 23655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } }
I20260924 00:55:23.187559 23655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9670171074484c6ea78d9efde06c1313, State: Initialized, Role: FOLLOWER
I20260924 00:55:23.188524 23655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } }
I20260924 00:55:23.191259 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.124806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.0.222.62:41621)
0924 00:55:23.125238 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.125245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.126229 (+ 984us) server_negotiation.cc:407] Connection header received
0924 00:55:23.134287 (+ 8058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.134293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.134386 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.134461 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.142305 (+ 7844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.143419 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.149826 (+ 6407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.150495 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.150616 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.182709 (+ 32093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:23.182716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:23.182722 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:23.182769 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:23.186963 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.188098 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.188104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.188108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.188265 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.188744 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.188749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.188752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.189223 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:23.189477 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.190692 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.190935 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260924 00:55:23.205746 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.124209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.222.61:44357)
0924 00:55:23.124421 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.124427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.162810 (+ 38383us) server_negotiation.cc:407] Connection header received
0924 00:55:23.163034 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.163039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.163119 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.163226 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.174531 (+ 11305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.175690 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.181518 (+ 5828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.182271 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.182417 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.197833 (+ 15416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:23.197841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:23.197847 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:23.197900 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:23.202135 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.203122 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.203129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.203133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.203336 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:23.203838 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:23.203844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:23.203848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:23.204256 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:23.204541 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.205106 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.205411 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:55:23.210503 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d23ad8fe91473a89e304cfc6511222. Latest consensus state: current_term: 1 leader_uuid: "41d23ad8fe91473a89e304cfc6511222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.210827 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:23.213555 23776 mvcc.cc:205] Tried to move back new op lower bound from 7332705579658616832 to 7332705579496992768. Current Snapshot: MvccSnapshot[applied={T|T < 7332705579658616832}]
I20260924 00:55:23.217115 23779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.217774 23779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:23.220220 23778 mvcc.cc:205] Tried to move back new op lower bound from 7332705579658616832 to 7332705579496992768. Current Snapshot: MvccSnapshot[applied={T|T < 7332705579658616832}]
I20260924 00:55:23.220904 23629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:23.221306 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41d23ad8fe91473a89e304cfc6511222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9670171074484c6ea78d9efde06c1313"
I20260924 00:55:23.225626 23629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [term 1 FOLLOWER]: Refusing update from remote peer 41d23ad8fe91473a89e304cfc6511222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:55:23.226922 23774 catalog_manager.cc:1347] Generated new cluster ID: 64fa5f26400c41e2a4065c86e3de6d06
I20260924 00:55:23.227029 23774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:23.228543 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41d23ad8fe91473a89e304cfc6511222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.228832 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:23.229141 23630 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41d23ad8fe91473a89e304cfc6511222 in term 1.
I20260924 00:55:23.229651 23783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41d23ad8fe91473a89e304cfc6511222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.230086 23783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:23.230988 23655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:23.233034 23784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:55:23.267923 23774 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:23.270133 23774 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:23.276787 23779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d23ad8fe91473a89e304cfc6511222. Latest consensus state: current_term: 1 leader_uuid: "41d23ad8fe91473a89e304cfc6511222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.277122 23779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:23.277244 23784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41d23ad8fe91473a89e304cfc6511222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.277627 23784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:23.280218 23789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41d23ad8fe91473a89e304cfc6511222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d23ad8fe91473a89e304cfc6511222" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 41621 } } peers { permanent_uuid: "9670171074484c6ea78d9efde06c1313" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 44357 } } peers { permanent_uuid: "06402b960c024908afa56558368d930f" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 32953 } } }
I20260924 00:55:23.281467 23789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:23.304365 23774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d23ad8fe91473a89e304cfc6511222: Generated new TSK 0
I20260924 00:55:23.305809 23774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:23.331048 23800 catalog_manager.cc:1259] Loaded cluster ID: 64fa5f26400c41e2a4065c86e3de6d06
I20260924 00:55:23.331163 23800 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313: loading cluster ID for follower catalog manager: success
I20260924 00:55:23.334870 23800 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313: acquiring CA information for follower catalog manager: success
I20260924 00:55:23.348798 23800 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9670171074484c6ea78d9efde06c1313: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 00:55:23.431890 23735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.420126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.0.222.62:41621)
0924 00:55:23.420335 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.420344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.429821 (+ 9477us) server_negotiation.cc:407] Connection header received
0924 00:55:23.430147 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.430153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.430241 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.430317 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.431594 (+ 1277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36468: BlockingRecv error: recv got EOF from 127.0.0.1:36468 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 00:55:23.434520 23736 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.422059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.222.61:44357)
0924 00:55:23.422242 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.422248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.432757 (+ 10509us) server_negotiation.cc:407] Connection header received
0924 00:55:23.433013 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.433023 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.433132 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.433223 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.434244 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41206: BlockingRecv error: recv got EOF from 127.0.0.1:41206 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 00:55:23.436936 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.430194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.0.222.60:32953)
0924 00:55:23.430410 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.430416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.435251 (+ 4835us) server_negotiation.cc:407] Connection header received
0924 00:55:23.435478 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.435489 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.435598 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.435667 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.436665 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58310: BlockingRecv error: recv got EOF from 127.0.0.1:58310 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 00:55:23.498339 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.474523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.0.222.61:44357)
0924 00:55:23.474734 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.474742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.489805 (+ 15063us) server_negotiation.cc:407] Connection header received
0924 00:55:23.490297 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.490304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.490579 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.490742 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:23.491615 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.492765 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.495132 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.495823 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.495929 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.496751 (+ 822us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:23.497515 (+ 764us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:23.497640 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.497879 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.497980 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 00:55:23.555833 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.541579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.0.222.60:32953)
0924 00:55:23.541820 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.541828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.553912 (+ 12084us) server_negotiation.cc:407] Connection header received
0924 00:55:23.554145 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.554151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.554290 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.554420 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.555565 (+ 1145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58318: BlockingRecv error: recv got EOF from 127.0.0.1:58318 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 00:55:23.619738 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.602017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.0.222.61:44357)
0924 00:55:23.602231 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.602238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.612693 (+ 10455us) server_negotiation.cc:407] Connection header received
0924 00:55:23.613255 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.613262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.613376 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.613441 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:23.614139 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.615244 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.617338 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.618073 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.618197 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.618844 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:23.619014 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:23.619075 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.619296 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.619387 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 00:55:23.647357 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.645926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58328 (local address 127.0.222.60:32953)
0924 00:55:23.646104 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.646110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.646129 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:55:23.646201 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.646206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.646271 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.646358 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.647128 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58328: BlockingRecv error: recv got EOF from 127.0.0.1:58328 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:55:23.660917 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.655669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.0.222.61:44357)
0924 00:55:23.655831 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.655837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.656228 (+ 391us) server_negotiation.cc:407] Connection header received
0924 00:55:23.656423 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.656428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.656497 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.656572 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:23.656988 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.657876 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.659234 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.659645 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.659751 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.660230 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:23.660337 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:23.660410 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.660558 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.660644 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 00:55:23.669018 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.667177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.0.222.60:32953)
0924 00:55:23.667384 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.667389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.667717 (+ 328us) server_negotiation.cc:407] Connection header received
0924 00:55:23.667914 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.667920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.668005 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.668093 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.668807 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58330: BlockingRecv error: recv got EOF from 127.0.0.1:58330 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:55:23.682427 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.676676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.0.222.61:44357)
0924 00:55:23.676901 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.676907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.677291 (+ 384us) server_negotiation.cc:407] Connection header received
0924 00:55:23.677503 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.677509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.677590 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.677667 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:23.678089 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.678938 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.680540 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.681050 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.681174 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.681669 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:23.681799 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:23.681887 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.682085 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.682170 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260924 00:55:23.690550 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.688150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.0.222.60:32953)
0924 00:55:23.688462 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.688469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.688997 (+ 528us) server_negotiation.cc:407] Connection header received
0924 00:55:23.689247 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.689254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.689340 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.689426 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.690165 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58346: BlockingRecv error: recv got EOF from 127.0.0.1:58346 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
I20260924 00:55:23.708189 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.702549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.0.222.61:44357)
0924 00:55:23.702782 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.702789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.703360 (+ 571us) server_negotiation.cc:407] Connection header received
0924 00:55:23.703564 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.703568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.703644 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.703728 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:23.704168 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.705010 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.706428 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.706892 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.707008 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.707414 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:23.707525 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:23.707598 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.707749 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.707835 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 00:55:23.725521 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.723340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58352 (local address 127.0.222.60:32953)
0924 00:55:23.723552 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.723559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.724064 (+ 505us) server_negotiation.cc:407] Connection header received
0924 00:55:23.724261 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.724266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.724344 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.724428 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.725264 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58352: BlockingRecv error: recv got EOF from 127.0.0.1:58352 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 00:55:23.750911 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.745177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.0.222.61:44357)
0924 00:55:23.745437 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.745444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.745989 (+ 545us) server_negotiation.cc:407] Connection header received
0924 00:55:23.746195 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.746199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.746276 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.746354 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:23.746799 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.747584 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.749129 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.749628 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.749754 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.750183 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:23.750295 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:23.750369 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.750522 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.750613 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260924 00:55:23.762084 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.757097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.0.222.60:32953)
0924 00:55:23.757331 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.757338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.760583 (+ 3245us) server_negotiation.cc:407] Connection header received
0924 00:55:23.760779 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.760783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.760857 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.760939 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.761826 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58356: BlockingRecv error: recv got EOF from 127.0.0.1:58356 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 00:55:23.802438 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.797070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.0.222.61:44357)
0924 00:55:23.797294 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.797301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.797567 (+ 266us) server_negotiation.cc:407] Connection header received
0924 00:55:23.797777 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.797782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.797855 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.797943 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:23.798356 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.799175 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.800512 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.800993 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.801125 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.801732 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:23.801843 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:23.801915 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.802064 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.802151 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 00:55:23.810284 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.808211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.0.222.60:32953)
0924 00:55:23.808413 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.808419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.808743 (+ 324us) server_negotiation.cc:407] Connection header received
0924 00:55:23.808892 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.808898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.809010 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.809132 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.810052 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58372: BlockingRecv error: recv got EOF from 127.0.0.1:58372 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:55:23.883008 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:23.877702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.0.222.61:44357)
0924 00:55:23.877935 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.877943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.878323 (+ 380us) server_negotiation.cc:407] Connection header received
0924 00:55:23.878523 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.878527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.878642 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.878719 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:23.879120 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.879996 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.881432 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:23.881876 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:23.881975 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:23.882364 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:23.882479 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:23.882545 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:23.882714 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 00:55:23.882780 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 00:55:23.890975 23732 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:23.888966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.0.222.60:32953)
0924 00:55:23.889171 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:23.889176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:23.889550 (+ 374us) server_negotiation.cc:407] Connection header received
0924 00:55:23.889771 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:23.889777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:23.889885 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:23.889994 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:23.890735 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58386: BlockingRecv error: recv got EOF from 127.0.0.1:58386 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:55:24.027367 23749 catalog_manager.cc:1259] Loaded cluster ID: 64fa5f26400c41e2a4065c86e3de6d06
I20260924 00:55:24.027493 23749 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: loading cluster ID for follower catalog manager: success
I20260924 00:55:24.029150 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:24.023336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.0.222.61:44357)
0924 00:55:24.023580 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:24.023587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:24.023861 (+ 274us) server_negotiation.cc:407] Connection header received
0924 00:55:24.024064 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:24.024068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:24.024138 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:24.024217 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:24.024567 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.025348 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.027183 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.027677 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.027784 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:24.028413 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:24.028527 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:24.028609 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:24.028814 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 00:55:24.028888 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 00:55:24.031028 23749 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: acquiring CA information for follower catalog manager: success
I20260924 00:55:24.033345 23749 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06402b960c024908afa56558368d930f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:55:24.041002 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:24.035445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.0.222.60:32953)
0924 00:55:24.035688 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:24.035695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:24.035960 (+ 265us) server_negotiation.cc:407] Connection header received
0924 00:55:24.036159 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:24.036165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:24.036298 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:24.036394 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:24.036809 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.037548 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.038847 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.039270 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.039379 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:24.039874 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:24.040412 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:24.040554 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:24.040652 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 00:55:24.040736 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 00:55:24.058166 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:24.051198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.0.222.61:44357)
0924 00:55:24.051398 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:24.051405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:24.052579 (+ 1174us) server_negotiation.cc:407] Connection header received
0924 00:55:24.052823 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:24.052827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:24.052902 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:24.052977 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:24.053548 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.054600 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.056476 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.056887 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.056998 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:24.057449 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:24.057553 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:24.057634 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:24.057786 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:55:24.057867 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:55:24.059719 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:24.052352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.0.222.60:32953)
0924 00:55:24.052567 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:24.052573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:24.053965 (+ 1392us) server_negotiation.cc:407] Connection header received
0924 00:55:24.054125 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:24.054130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:24.054210 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:24.054299 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:24.054914 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.055598 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.057717 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.058268 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.058364 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:24.058975 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:24.059112 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:24.059208 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:24.059356 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 00:55:24.059439 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:55:24.060798 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:24.051043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.0.222.62:41621)
0924 00:55:24.053582 (+ 2539us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:24.053588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:24.053606 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:55:24.053734 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:24.053739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:24.053946 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:24.054114 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:24.054704 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.055852 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.058187 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:24.058650 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:24.058766 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:24.059739 (+ 973us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:24.060190 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:24.060300 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:24.060417 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 00:55:24.060501 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2414,"thread_start_us":2242,"threads_started":1}
I20260924 00:55:24.066699 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23514
I20260924 00:55:24.067799 23576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:24.180688 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 23514
I20260924 00:55:24.198490 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23588
I20260924 00:55:24.199658 23650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:24.306094 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 23588
I20260924 00:55:24.322598 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23662
I20260924 00:55:24.323966 23724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:24.431943 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 23662
2026-09-24T00:55:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4247 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:55:24 dist-test-slave-rtvz krb5kdc[23915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:55:24 dist-test-slave-rtvz krb5kdc[23915](info): set up 2 sockets
Sep 24 00:55:24 dist-test-slave-rtvz krb5kdc[23915](info): commencing operation
krb5kdc: starting...
W20260924 00:55:25.469300 888 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/test-user.keytab.
Sep 24 00:55:25 dist-test-slave-rtvz krb5kdc[23915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.222.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
Entry for principal kudu/127.0.222.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab.
I20260924 00:55:25.729655 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46199
--webserver_interface=127.0.222.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:46199,127.0.222.61:43725,127.0.222.60:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:55:26.064284 23928 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:55:26.064613 23928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:26.064705 23928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:26.064821 23928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:26.072214 23928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:26.072324 23928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:26.072386 23928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:26.072455 23928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:26.072517 23928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:26.083144 23928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:46199,127.0.222.61:43725,127.0.222.60:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.62.keytab
--principal=kudu/127.0.222.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.62:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.222.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:26.084482 23928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:26.085888 23928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:26.097743 23936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:26.097844 23934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:26.098035 23933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:26.098483 23928 server_base.cc:1034] running on GCE node
I20260924 00:55:26.099303 23928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:26.100831 23928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:26.102101 23928 hybrid_clock.cc:648] HybridClock initialized: now 1790211326102018 us; error 50 us; skew 500 ppm
I20260924 00:55:26.102478 23928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:26 dist-test-slave-rtvz krb5kdc[23915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:26.111269 23928 init.cc:377] Logged in from keytab as kudu/127.0.222.62@KRBTEST.COM (short username kudu)
I20260924 00:55:26.113047 23928 webserver.cc:460] Webserver started at http://127.0.222.62:34239/ using document root <none> and password file <none>
I20260924 00:55:26.113857 23928 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:26.113976 23928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:26.114300 23928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:26.117489 23928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/data/instance:
uuid: "b613217c44a549159de6051e96ed05c5"
format_stamp: "Formatted at 2026-09-24 00:55:26 on dist-test-slave-rtvz"
server_key: "275bcce2d64562edad6d375d195a1b1e"
server_key_iv: "83f0d2a88f0275bc054329f56c2da49e"
server_key_version: "encryptionkey@0"
I20260924 00:55:26.118312 23928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/wal/instance:
uuid: "b613217c44a549159de6051e96ed05c5"
format_stamp: "Formatted at 2026-09-24 00:55:26 on dist-test-slave-rtvz"
server_key: "275bcce2d64562edad6d375d195a1b1e"
server_key_iv: "83f0d2a88f0275bc054329f56c2da49e"
server_key_version: "encryptionkey@0"
I20260924 00:55:26.123919 23928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 00:55:26.127882 23943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:26.129388 23928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 00:55:26.129578 23928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/wal
uuid: "b613217c44a549159de6051e96ed05c5"
format_stamp: "Formatted at 2026-09-24 00:55:26 on dist-test-slave-rtvz"
server_key: "275bcce2d64562edad6d375d195a1b1e"
server_key_iv: "83f0d2a88f0275bc054329f56c2da49e"
server_key_version: "encryptionkey@0"
I20260924 00:55:26.129802 23928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:26.153016 23928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:26.157469 23928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:26.157735 23928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:26.175597 23928 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.62:46199
I20260924 00:55:26.175617 23994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.62:46199 every 8 connection(s)
I20260924 00:55:26.176970 23928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/data/info.pb
I20260924 00:55:26.180689 23995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:26.185184 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 23928
I20260924 00:55:26.185392 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-0/wal/instance
I20260924 00:55:26.185941 888 external_mini_cluster.cc:1428] Setting key 0d71e6c8fc6f48c787471d7733703134
I20260924 00:55:26.188180 23995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:26 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.61@KRBTEST.COM; defaulting to no policy
I20260924 00:55:26.212024 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.192051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46199 (local address 127.0.0.1:57344)
0924 00:55:26.193416 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.193468 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:26.195283 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:26.196314 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:26.196321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:26.197137 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:26.197629 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:26.197653 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.199239 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.199244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:26.200811 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:26.200826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.201840 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.201850 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:26.202010 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:26.202642 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:26.202682 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:26.205170 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:26.208241 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:26.208253 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:26.208266 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:26.208652 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:26.209032 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:26.209037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:26.209041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:26.209226 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:26.209956 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:26.209964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:26.210175 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 00:55:26.210389 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 00:55:26.211097 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":189,"spinlock_wait_cycles":36224,"thread_start_us":831,"threads_started":1}
I20260924 00:55:26.212045 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.192051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.0.222.62:46199)
0924 00:55:26.193041 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:26.193060 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:26.193604 (+ 544us) server_negotiation.cc:407] Connection header received
0924 00:55:26.195689 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:26.195698 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:26.196042 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:26.196281 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:26.197843 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.199078 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.200949 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.201605 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.201735 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:26.205398 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:26.205407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:26.205420 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:26.205452 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:26.208122 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:26.208777 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:26.208783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:26.208785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:26.208931 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:26.209318 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:26.209322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:26.209325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:26.209828 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:26.210011 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:26.210547 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 00:55:26.210905 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":740,"thread_start_us":473,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.61@KRBTEST.COM" created.
I20260924 00:55:26.215022 23995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:55:26.216127 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.215727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42456)
0924 00:55:26.215894 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.215998 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 00:55:26.216365 23947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
I20260924 00:55:26.217813 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.217487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42460)
0924 00:55:26.217618 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.217686 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 00:55:26.217998 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab.
Entry for principal kudu/127.0.222.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab.
I20260924 00:55:26.244971 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:43725
--webserver_interface=127.0.222.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:46199,127.0.222.61:43725,127.0.222.60:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:55:26.267331 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } attempt: 1
I20260924 00:55:26.268539 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.268066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42464)
0924 00:55:26.268275 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.268388 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 00:55:26.269665 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.269144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42474)
0924 00:55:26.269391 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.269494 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 00:55:26.269846 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
I20260924 00:55:26.338083 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } attempt: 2
I20260924 00:55:26.339299 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.338825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42484)
0924 00:55:26.339017 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.339121 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 00:55:26.340365 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.339995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42500)
0924 00:55:26.340144 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.340224 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 00:55:26.340557 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
I20260924 00:55:26.431804 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } attempt: 3
I20260924 00:55:26.433049 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.432609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42514)
0924 00:55:26.432810 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.432908 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 00:55:26.434258 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.433885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42518)
0924 00:55:26.434032 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.434110 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 00:55:26.434453 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
W20260924 00:55:26.575799 24002 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:55:26.576176 24002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:26.576277 24002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:26.576399 24002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 00:55:26.577726 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } attempt: 4
I20260924 00:55:26.578974 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.578425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42528)
0924 00:55:26.578755 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.578847 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260924 00:55:26.580024 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.579669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:42534)
0924 00:55:26.579815 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.579899 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 00:55:26.580204 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.61:43725: Network error: Client connection negotiation failed: client connection to 127.0.222.61:43725: connect: Connection refused (error 111)
W20260924 00:55:26.584264 24002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:26.584344 24002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:26.584414 24002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:26.584482 24002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:26.584558 24002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:26.594731 24002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:46199,127.0.222.61:43725,127.0.222.60:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.61.keytab
--principal=kudu/127.0.222.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.61:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.222.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:26.596073 24002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:26.597451 24002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:55:26.609093 24008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:26.609118 24010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:26.609041 24007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:26.610257 24002 server_base.cc:1034] running on GCE node
I20260924 00:55:26.611032 24002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:26.612486 24002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:26.613706 24002 hybrid_clock.cc:648] HybridClock initialized: now 1790211326613617 us; error 49 us; skew 500 ppm
I20260924 00:55:26.614017 24002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:26 dist-test-slave-rtvz krb5kdc[23915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:26.617432 24002 init.cc:377] Logged in from keytab as kudu/127.0.222.61@KRBTEST.COM (short username kudu)
I20260924 00:55:26.618966 24002 webserver.cc:460] Webserver started at http://127.0.222.61:34339/ using document root <none> and password file <none>
I20260924 00:55:26.619647 24002 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:26.619737 24002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:26.620029 24002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:26.623025 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/data/instance:
uuid: "1ad38d3f99b14d6bb6005775a83c3ad7"
format_stamp: "Formatted at 2026-09-24 00:55:26 on dist-test-slave-rtvz"
server_key: "bb6a7cc5de241d9050aa92598645190a"
server_key_iv: "f14a3ddc0e7364b315771ad7cf0533ba"
server_key_version: "encryptionkey@0"
I20260924 00:55:26.623755 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/wal/instance:
uuid: "1ad38d3f99b14d6bb6005775a83c3ad7"
format_stamp: "Formatted at 2026-09-24 00:55:26 on dist-test-slave-rtvz"
server_key: "bb6a7cc5de241d9050aa92598645190a"
server_key_iv: "f14a3ddc0e7364b315771ad7cf0533ba"
server_key_version: "encryptionkey@0"
I20260924 00:55:26.628855 24002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 00:55:26.632459 24017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:26.633988 24002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 00:55:26.634184 24002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/wal
uuid: "1ad38d3f99b14d6bb6005775a83c3ad7"
format_stamp: "Formatted at 2026-09-24 00:55:26 on dist-test-slave-rtvz"
server_key: "bb6a7cc5de241d9050aa92598645190a"
server_key_iv: "f14a3ddc0e7364b315771ad7cf0533ba"
server_key_version: "encryptionkey@0"
I20260924 00:55:26.634380 24002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:26.663647 24002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:26.668004 24002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:26.668282 24002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:26.685643 24002 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.61:43725
I20260924 00:55:26.685681 24068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.61:43725 every 8 connection(s)
I20260924 00:55:26.686969 24002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/data/info.pb
I20260924 00:55:26.690176 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 24002
I20260924 00:55:26.690379 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-1/wal/instance
I20260924 00:55:26.690915 24069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:26.690944 888 external_mini_cluster.cc:1428] Setting key 914056eff40e37ba7a80b873ac6f3320
I20260924 00:55:26.699646 24069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:26 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.222.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.222.60@KRBTEST.COM" created.
I20260924 00:55:26.721541 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.703154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.0.222.62:46199)
0924 00:55:26.703331 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:26.703335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:26.704945 (+ 1610us) server_negotiation.cc:407] Connection header received
0924 00:55:26.706961 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:26.706968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:26.707073 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:26.707163 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:26.708806 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.709550 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.711189 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.711628 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.711742 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:26.715394 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:26.715400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:26.715403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:26.715438 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:26.718292 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:26.719139 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:26.719145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:26.719149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:26.719289 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:26.719740 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:26.719744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:26.719746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:26.720041 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:26.720162 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:26.721061 (+ 899us) server_negotiation.cc:299] Negotiation successful
0924 00:55:26.721295 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:55:26.722905 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.703908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46199 (local address 127.0.0.1:56938)
0924 00:55:26.704772 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.704813 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:26.706662 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:26.707437 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:26.707446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:26.708016 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:26.708618 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:26.708637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.709714 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.709718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:26.711020 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:26.711035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.711874 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.711881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:26.712157 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:26.712812 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:26.712840 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:26.715159 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:26.718439 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:26.718449 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:26.718475 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:26.719004 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:26.719397 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:26.719402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:26.719406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:26.719618 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:26.720174 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:26.720182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:26.720675 (+ 493us) client_negotiation.cc:769] Sending connection context
0924 00:55:26.720985 (+ 310us) client_negotiation.cc:240] Negotiation successful
0924 00:55:26.721568 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":361,"threads_started":1}
I20260924 00:55:26.724668 24069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:26 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:55:26.740221 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.725513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:45942)
0924 00:55:26.725688 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.725707 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:26.725808 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:26.726918 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:26.726922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:26.727292 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:26.727438 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:26.727446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.729254 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.729259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:26.730365 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:26.730380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.731193 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.731203 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:26.731369 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:26.731954 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:26.731975 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:26.734495 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:26.737629 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:26.737634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:26.737638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:26.738051 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:26.738490 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:26.738494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:26.738496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:26.738658 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:26.739239 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:26.739243 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:26.739380 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 00:55:26.739488 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 00:55:26.739767 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 00:55:26.740262 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.725739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.0.222.61:43725)
0924 00:55:26.726382 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:26.726389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:26.726418 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:55:26.726509 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:26.726518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:26.726759 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:26.726898 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:26.727602 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.729085 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.730525 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.731052 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.731145 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:26.734741 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:26.734754 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:26.734770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:26.734804 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:26.737509 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:26.738176 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:26.738183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:26.738185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:26.738311 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:26.738767 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:26.738771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:26.738773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:26.739117 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:26.739318 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:26.739684 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 00:55:26.739893 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":340,"threads_started":1}
I20260924 00:55:26.742954 24069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:55:26.744202 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.743696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54384)
0924 00:55:26.743867 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.744000 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 00:55:26.744481 24018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
I20260924 00:55:26.746133 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.745792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54390)
0924 00:55:26.745934 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.746012 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 00:55:26.746344 24069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab.
Entry for principal kudu/127.0.222.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab.
I20260924 00:55:26.752900 888 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfVll0B/build/asan/bin/kudu
/tmp/dist-test-taskfVll0B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:37181
--webserver_interface=127.0.222.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.222.20:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.222.62:46199,127.0.222.61:43725,127.0.222.60:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:55:26.795681 24069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 1
I20260924 00:55:26.796902 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.796432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54406)
0924 00:55:26.796639 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.796747 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 00:55:26.798077 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.797718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54422)
0924 00:55:26.797877 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.797954 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 00:55:26.798293 24069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
I20260924 00:55:26.866528 24069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 2
I20260924 00:55:26.867599 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.867144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54428)
0924 00:55:26.867340 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.867430 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 00:55:26.868567 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.868252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54444)
0924 00:55:26.868388 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.868454 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 00:55:26.868762 24069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
I20260924 00:55:26.879453 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } attempt: 5
Sep 24 00:55:26 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:55:26.895103 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.880190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:45958)
0924 00:55:26.880394 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.880412 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:26.880539 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:26.881003 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:26.881009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:26.881647 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:26.881860 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:26.881871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.883286 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.883291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:26.884271 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:26.884285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.885354 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.885359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:26.885499 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:26.886097 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:26.886116 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:26.888137 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:26.892278 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:26.892285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:26.892289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:26.892810 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:26.893303 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:26.893308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:26.893310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:26.893473 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:26.894265 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:26.894269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:26.894428 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 00:55:26.894583 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 00:55:26.894837 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 00:55:26.895270 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.880334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.0.222.61:43725)
0924 00:55:26.880557 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:26.880562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:26.880586 (+ 24us) server_negotiation.cc:407] Connection header received
0924 00:55:26.880656 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:26.880662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:26.880763 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:26.880884 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:26.882029 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.883125 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.884454 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:26.885150 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:26.885323 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:26.888400 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:26.888408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:26.888413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:26.888463 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:26.892053 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:26.892955 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:26.892961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:26.892965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:26.893127 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:26.893656 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:26.893662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:26.893665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:26.894067 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:26.894213 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:26.894715 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 00:55:26.894956 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 00:55:26.896430 23995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:55:26.897647 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.897253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54458)
0924 00:55:26.897416 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.897497 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 00:55:26.898653 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.898328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54468)
0924 00:55:26.898461 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.898549 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 00:55:26.898880 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
I20260924 00:55:26.950109 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 1
I20260924 00:55:26.951327 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.950869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54482)
0924 00:55:26.951063 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.951159 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 00:55:26.952344 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.952006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54490)
0924 00:55:26.952153 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.952226 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 00:55:26.952558 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
I20260924 00:55:26.959995 24069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 3
I20260924 00:55:26.961225 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.960657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54500)
0924 00:55:26.960872 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.960960 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 00:55:26.962280 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:26.961823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54510)
0924 00:55:26.962081 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:26.962162 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260924 00:55:26.962486 24069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
I20260924 00:55:27.020826 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 2
I20260924 00:55:27.022233 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.021671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54518)
0924 00:55:27.021900 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.022058 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 00:55:27.023301 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.022936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54520)
0924 00:55:27.023112 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.023184 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 00:55:27.023542 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
W20260924 00:55:27.095928 24076 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:55:27.096292 24076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:55:27.096388 24076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:55:27.096503 24076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:55:27.103994 24076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:55:27.104149 24076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:55:27.104256 24076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:55:27.104327 24076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:55:27.104417 24076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:55:27.105679 24069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 4
I20260924 00:55:27.106815 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.106320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54524)
0924 00:55:27.106554 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.106656 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 00:55:27.107831 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.107429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54526)
0924 00:55:27.107602 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.107704 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 00:55:27.108037 24069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
I20260924 00:55:27.114912 24076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.222.20:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.222.62:46199,127.0.222.61:43725,127.0.222.60:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/krb5kdc/kudu_127.0.222.60.keytab
--principal=kudu/127.0.222.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.222.60:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.222.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 23:51:06 UTC on ccycloud.quasar-ikldij.root.comops.site
build id 4669120
ASAN enabled
I20260924 00:55:27.116349 24076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:55:27.117710 24076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:55:27.129732 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 3
W20260924 00:55:27.129822 24082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:27.129727 24084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:55:27.130003 24081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:55:27.130812 24076 server_base.cc:1034] running on GCE node
I20260924 00:55:27.130931 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.130442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54530)
0924 00:55:27.130624 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.130725 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 00:55:27.131755 24076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:55:27.132119 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.131664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54546)
0924 00:55:27.131877 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.131964 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 00:55:27.132385 23995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.222.60:37181: Network error: Client connection negotiation failed: client connection to 127.0.222.60:37181: connect: Connection refused (error 111)
I20260924 00:55:27.133445 24076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:55:27.134706 24076 hybrid_clock.cc:648] HybridClock initialized: now 1790211327134622 us; error 54 us; skew 500 ppm
I20260924 00:55:27.135044 24076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:55:27.138295 24076 init.cc:377] Logged in from keytab as kudu/127.0.222.60@KRBTEST.COM (short username kudu)
I20260924 00:55:27.139742 24076 webserver.cc:460] Webserver started at http://127.0.222.60:38579/ using document root <none> and password file <none>
I20260924 00:55:27.140444 24076 fs_manager.cc:359] Metadata directory not provided
I20260924 00:55:27.140547 24076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:55:27.140854 24076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:55:27.144016 24076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/data/instance:
uuid: "5f6969f80d344471a170cf800dc581a6"
format_stamp: "Formatted at 2026-09-24 00:55:27 on dist-test-slave-rtvz"
server_key: "d84bf7669ef255ebe97d00a974c493ea"
server_key_iv: "69a7eb6ec4cb547ed7e306266f1cdba2"
server_key_version: "encryptionkey@0"
I20260924 00:55:27.144852 24076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/wal/instance:
uuid: "5f6969f80d344471a170cf800dc581a6"
format_stamp: "Formatted at 2026-09-24 00:55:27 on dist-test-slave-rtvz"
server_key: "d84bf7669ef255ebe97d00a974c493ea"
server_key_iv: "69a7eb6ec4cb547ed7e306266f1cdba2"
server_key_version: "encryptionkey@0"
I20260924 00:55:27.150547 24076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 00:55:27.154337 24091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:55:27.155792 24076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 00:55:27.155965 24076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/data,/tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/wal
uuid: "5f6969f80d344471a170cf800dc581a6"
format_stamp: "Formatted at 2026-09-24 00:55:27 on dist-test-slave-rtvz"
server_key: "d84bf7669ef255ebe97d00a974c493ea"
server_key_iv: "69a7eb6ec4cb547ed7e306266f1cdba2"
server_key_version: "encryptionkey@0"
I20260924 00:55:27.156186 24076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:55:27.175140 24076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:55:27.179425 24076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:55:27.179682 24076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:55:27.197676 24076 rpc_server.cc:307] RPC server started. Bound to: 127.0.222.60:37181
I20260924 00:55:27.197726 24142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.222.60:37181 every 8 connection(s)
I20260924 00:55:27.199019 24076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/data/info.pb
I20260924 00:55:27.202770 24143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:55:27.208073 888 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfVll0B/build/asan/bin/kudu as pid 24076
I20260924 00:55:27.208294 888 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfVll0B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790210962623352-888-0/minicluster-data/master-2/wal/instance
I20260924 00:55:27.208707 888 external_mini_cluster.cc:1428] Setting key f261dd4cb4d87fc1c3572a835eeeb9c0
I20260924 00:55:27.210374 24143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260924 00:55:27.229529 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.213299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.0.222.62:46199)
0924 00:55:27.213475 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.213482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.214063 (+ 581us) server_negotiation.cc:407] Connection header received
0924 00:55:27.214297 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.214303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.214385 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.214478 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.215782 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.216519 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.218263 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.218785 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.218880 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.222791 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.222798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.222803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.222849 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.225888 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.226941 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.226945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.226948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.227108 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.227568 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.227571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.227573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.227876 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.228012 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.228852 (+ 840us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.229108 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:55:27.230706 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.210432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.0.222.60:37181)
0924 00:55:27.211260 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.211275 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.211327 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:55:27.213358 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.213399 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.214021 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.214338 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.215819 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.217156 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.218490 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.219141 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.219380 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.223774 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.223783 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.223798 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.223829 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.227020 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.227753 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.227761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.227763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.227889 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.228279 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.228284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.228287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.228664 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.228958 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.229336 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.229591 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":606,"thread_start_us":386,"threads_started":1}
I20260924 00:55:27.230743 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.213163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.62:46199 (local address 127.0.0.1:56952)
0924 00:55:27.213914 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.213953 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:27.214124 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:27.214566 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:27.214572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:27.215226 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:27.215593 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.215613 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.216685 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.216691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.218113 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.218125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.219012 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.219020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.219375 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.220153 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:27.220184 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:27.222543 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:27.226060 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.226079 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.226095 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.226778 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.227248 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.227252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.227254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.227442 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.228010 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:27.228018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:27.228418 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 00:55:27.228710 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 00:55:27.229301 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":9,"thread_start_us":365,"threads_started":1}
I20260924 00:55:27.232403 24143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.62@KRBTEST.COM
I20260924 00:55:27.247264 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.233338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.0.222.61:43725)
0924 00:55:27.233513 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.233517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.233533 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:55:27.233610 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.233614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.233677 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.233784 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.234692 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.235764 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.237478 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.237922 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.238029 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.241642 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.241647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.241650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.241682 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.244696 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.245422 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.245426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.245429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.245574 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.245932 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.245935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.245938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.246237 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.246378 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.246784 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.246954 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 00:55:27.247309 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.233234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.61:43725 (local address 127.0.0.1:45970)
0924 00:55:27.233388 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.233408 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:27.233509 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:27.233823 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:27.233828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:27.234261 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:27.234525 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.234537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.235905 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.235909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.237347 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.237359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.238054 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.238060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.238261 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.238814 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:27.238832 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:27.241453 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:27.244848 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.244853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.244856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.245300 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.245684 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.245688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.245690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.245823 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.246357 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:27.246362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:27.246523 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 00:55:27.246667 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 00:55:27.246963 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 00:55:27.248363 24143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:55:27.249291 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.234141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.0.222.62:46199)
0924 00:55:27.234297 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.234303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.234320 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:55:27.234478 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.234483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.234555 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.234650 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.236081 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.237167 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.238583 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.239059 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.239164 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.243639 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.243647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.243651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.243696 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.246412 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.247231 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.247237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.247240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.247408 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.247869 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.247875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.247879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.248158 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.248420 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.248744 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.248958 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.60@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.222.61@KRBTEST.COM
I20260924 00:55:27.261585 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.249320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.0.222.60:37181)
0924 00:55:27.249542 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.249546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.249561 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:55:27.249691 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.249694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.249751 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.249816 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.250551 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.251474 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.252569 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.252990 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.253096 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.256108 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.256114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.256117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.256147 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.258811 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.259487 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.259493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.259496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.259662 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.259994 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.259998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.260001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.260382 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.260523 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.260961 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.261307 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":124}
I20260924 00:55:27.261616 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.249059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54560)
0924 00:55:27.249464 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.249490 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:27.249584 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:27.249869 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:27.249874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:27.250239 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:27.250418 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.250430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.251582 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.251587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.252445 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.252455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.253124 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.253129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.253354 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.253874 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:27.253892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:27.255937 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:27.258959 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.258963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.258966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.259349 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.259768 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.259771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.259774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.259896 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.260567 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:27.260570 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:27.260728 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 00:55:27.260851 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 00:55:27.261148 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308}
I20260924 00:55:27.266027 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.251217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.0.222.61:43725)
0924 00:55:27.251361 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.251365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.251380 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:55:27.251566 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.251570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.251634 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.251745 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.253009 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.253779 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.254943 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.255421 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.255556 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.259927 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.259932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.259935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.259976 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.263327 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.264047 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.264054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.264058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.264215 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.264601 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.264607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.264611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.264892 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.265173 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.265478 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.265722 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:55:27.266621 24143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: Bootstrap starting.
I20260924 00:55:27.270566 24143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:27.271008 888 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:55:27.271880 24143 log.cc:824] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:27.274987 24143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: No bootstrap required, opened a new log
I20260924 00:55:27.274029 23970 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b613217c44a549159de6051e96ed05c5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56966
I20260924 00:55:27.282223 24143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } }
I20260924 00:55:27.282564 24143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6969f80d344471a170cf800dc581a6, State: Initialized, Role: FOLLOWER
I20260924 00:55:27.283015 24143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } }
I20260924 00:55:27.283975 24149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.284392 24149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:27.284875 24143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 00:55:27.293911 24160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:55:27.294051 24160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:55:27.291422 24076 master_runner.cc:186] Error getting master registration for 127.0.222.62:46199: OK, instance_id { permanent_uuid: "b613217c44a549159de6051e96ed05c5" instance_seqno: 1790211326170471 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:55:27.301964 24076 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:55:27.309600 23995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 4
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.62@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260924 00:55:27.326476 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.310426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.0.222.60:37181)
0924 00:55:27.310628 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.310639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.310663 (+ 24us) server_negotiation.cc:407] Connection header received
0924 00:55:27.310801 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.310806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.310894 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.311008 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.311937 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.313082 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.314923 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.315670 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.315804 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.319771 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.319778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.319783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.319828 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.323581 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.324309 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.324315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.324319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.324485 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.324882 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.324887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.324891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.325329 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.325489 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.325876 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.326115 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 00:55:27.326473 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.310352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54570)
0924 00:55:27.310546 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.310565 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:27.310682 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:27.311051 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:27.311056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:27.311565 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:27.311769 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.311781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.313252 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.313258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.314754 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.314772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.315816 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.315821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.316035 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.316719 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:27.316741 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:27.319521 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:27.323754 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.323760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.323764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.324179 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.324610 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.324615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.324619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.324771 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.325449 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:27.325454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:27.325634 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 00:55:27.325758 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 00:55:27.326087 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 00:55:27.331485 23995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5: Bootstrap starting.
I20260924 00:55:27.335659 23995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:27.336944 23995 log.cc:824] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:27.340371 23995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5: No bootstrap required, opened a new log
I20260924 00:55:27.348124 23995 raft_consensus.cc:348] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } }
I20260924 00:55:27.348608 23995 raft_consensus.cc:729] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b613217c44a549159de6051e96ed05c5, State: Initialized, Role: FOLLOWER
I20260924 00:55:27.349156 23995 consensus_queue.cc:260] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } }
I20260924 00:55:27.350427 24163 sys_catalog.cc:455] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.351052 24163 sys_catalog.cc:458] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:27.351755 23995 sys_catalog.cc:564] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:27.351856 23970 raft_consensus.cc:482] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:55:27.352057 23970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:27.355955 23970 raft_consensus.cc:504] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } }
I20260924 00:55:27.359757 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b613217c44a549159de6051e96ed05c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5f6969f80d344471a170cf800dc581a6"
I20260924 00:55:27.360383 24118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:27.365513 24118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b613217c44a549159de6051e96ed05c5 in term 1.
I20260924 00:55:27.366230 23944 leader_election.cc:302] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f6969f80d344471a170cf800dc581a6, b613217c44a549159de6051e96ed05c5; no voters:
I20260924 00:55:27.366809 23970 leader_election.cc:288] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [CANDIDATE]: Term 1 election: Requested vote from peers 1ad38d3f99b14d6bb6005775a83c3ad7 (127.0.222.61:43725), 5f6969f80d344471a170cf800dc581a6 (127.0.222.60:37181)
I20260924 00:55:27.373525 24163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:55:27.366501 23928 master_runner.cc:186] Error getting master registration for 127.0.222.61:43725: OK, instance_id { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" instance_seqno: 1790211326680624 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:55:27.377499 23928 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:55:27.379953 24163 raft_consensus.cc:686] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [term 1 LEADER]: Becoming Leader. State: Replica: b613217c44a549159de6051e96ed05c5, State: Running, Role: LEADER
I20260924 00:55:27.380579 24163 consensus_queue.cc:237] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } }
I20260924 00:55:27.385393 24177 sys_catalog.cc:455] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b613217c44a549159de6051e96ed05c5. Latest consensus state: current_term: 1 leader_uuid: "b613217c44a549159de6051e96ed05c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.385874 24177 sys_catalog.cc:458] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: This master's current role is: LEADER
W20260924 00:55:27.389842 24174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:55:27.389957 24174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:55:27.390547 24184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:55:27.392488 24184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:55:27.407014 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.380494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.0.222.60:37181)
0924 00:55:27.380831 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.380836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.386769 (+ 5933us) server_negotiation.cc:407] Connection header received
0924 00:55:27.389543 (+ 2774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.389551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.389678 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.389759 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.392191 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.393460 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.395457 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.396129 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.396228 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.400037 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.400045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.400050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.400096 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.403796 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.404589 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.404595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.404599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.404770 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.405231 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.405236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.405239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.405582 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.405814 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.406383 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.406623 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260924 00:55:27.407407 24069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } attempt: 5
W20260924 00:55:27.410802 23949 tablet.cc:2369] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 00:55:27 dist-test-slave-rtvz krb5kdc[23915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790211326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.222.61@KRBTEST.COM for kudu/127.0.222.60@KRBTEST.COM
I20260924 00:55:27.423007 24118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [term 1 FOLLOWER]: Refusing update from remote peer b613217c44a549159de6051e96ed05c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:55:27.424264 24163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:55:27.426144 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.388734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.0.222.61:43725)
0924 00:55:27.388921 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.388927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.398594 (+ 9667us) server_negotiation.cc:407] Connection header received
0924 00:55:27.398864 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.398870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.398956 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.399027 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.404133 (+ 5106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.405296 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.409542 (+ 4246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.411644 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.411769 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.414824 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.414830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.414835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.414882 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.420686 (+ 5804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.422164 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.422171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.422174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.422436 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.423189 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.423194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.423218 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.423830 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.424202 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.425240 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.425639 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 00:55:27.431885 24097 tablet.cc:2369] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:55:27.440313 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.408003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.222.60:37181 (local address 127.0.0.1:54588)
0924 00:55:27.408256 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 00:55:27.408274 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:55:27.408397 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:55:27.408760 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:55:27.408765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:55:27.409327 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:55:27.410993 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.411008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.413336 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.413342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.414731 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:55:27.414746 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.415754 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.415760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:55:27.416000 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.416557 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:55:27.416583 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:55:27.429669 (+ 13086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:55:27.437346 (+ 7677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.437352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.437356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.437930 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.438447 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:55:27.438452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:55:27.438455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:55:27.438678 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:55:27.439313 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:55:27.439317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:55:27.439485 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 00:55:27.439618 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 00:55:27.439926 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260924 00:55:27.441377 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.408097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.0.222.60:37181)
0924 00:55:27.408256 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.408261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.408330 (+ 69us) server_negotiation.cc:407] Connection header received
0924 00:55:27.408514 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.408518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.408591 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.411931 (+ 3340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.412078 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.413139 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.414900 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.415578 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.416300 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.431002 (+ 14702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.431013 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.431019 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.431063 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.435339 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.438107 (+ 2768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.438112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.438115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.438266 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.438819 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.438824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.438828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.439178 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.440570 (+ 1392us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.440728 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.441003 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:55:27.441422 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.380306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.222.62:46199)
0924 00:55:27.380686 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.380692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.395197 (+ 14505us) server_negotiation.cc:407] Connection header received
0924 00:55:27.395643 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.395649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.395729 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.395793 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.400801 (+ 5008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.401952 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.407378 (+ 5426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.408136 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.408239 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.432766 (+ 24527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:55:27.432775 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:55:27.432779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:55:27.432849 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:55:27.436568 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.437932 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.437938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.437941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.438133 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:55:27.438708 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:55:27.438713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:55:27.438717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:55:27.439142 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:55:27.439418 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.440051 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.440745 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260924 00:55:27.446789 24149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b613217c44a549159de6051e96ed05c5. Latest consensus state: current_term: 1 leader_uuid: "b613217c44a549159de6051e96ed05c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.447134 24069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7: Bootstrap starting.
I20260924 00:55:27.447175 24149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:27.452442 24069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7: Neither blocks nor log segments found. Creating new log.
I20260924 00:55:27.454253 24186 mvcc.cc:205] Tried to move back new op lower bound from 7332705597103218688 to 7332705596957114368. Current Snapshot: MvccSnapshot[applied={T|T < 7332705597103218688}]
I20260924 00:55:27.454792 24187 mvcc.cc:205] Tried to move back new op lower bound from 7332705597103218688 to 7332705596957114368. Current Snapshot: MvccSnapshot[applied={T|T < 7332705597103218688}]
I20260924 00:55:27.457667 24069 log.cc:824] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:55:27.458412 24163 sys_catalog.cc:455] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b613217c44a549159de6051e96ed05c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.458702 24163 sys_catalog.cc:458] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:27.464174 24184 catalog_manager.cc:1347] Generated new cluster ID: 756be449629e4572b76e122e51c31ae3
I20260924 00:55:27.464284 24184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:55:27.465642 24069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7: No bootstrap required, opened a new log
I20260924 00:55:27.482499 24149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b613217c44a549159de6051e96ed05c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.482815 24149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:27.482972 24069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } }
I20260924 00:55:27.483433 24069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad38d3f99b14d6bb6005775a83c3ad7, State: Initialized, Role: FOLLOWER
I20260924 00:55:27.484045 24069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } }
I20260924 00:55:27.493782 24192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.494238 24192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:27.496634 24069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:55:27.497059 24043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:55:27.497498 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b613217c44a549159de6051e96ed05c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7"
I20260924 00:55:27.501691 24043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [term 1 FOLLOWER]: Refusing update from remote peer b613217c44a549159de6051e96ed05c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:55:27.505064 24044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b613217c44a549159de6051e96ed05c5 in term 1.
I20260924 00:55:27.507136 24177 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:55:27.531685 24184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:55:27.533790 24184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:55:27.549860 24192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b613217c44a549159de6051e96ed05c5. Latest consensus state: current_term: 1 leader_uuid: "b613217c44a549159de6051e96ed05c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.550211 24192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:55:27.550597 24163 sys_catalog.cc:455] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b613217c44a549159de6051e96ed05c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.550921 24163 sys_catalog.cc:458] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5 [sys.catalog]: This master's current role is: LEADER
I20260924 00:55:27.562614 24184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b613217c44a549159de6051e96ed05c5: Generated new TSK 0
I20260924 00:55:27.563902 24184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:55:27.567909 24197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b613217c44a549159de6051e96ed05c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b613217c44a549159de6051e96ed05c5" member_type: VOTER last_known_addr { host: "127.0.222.62" port: 46199 } } peers { permanent_uuid: "1ad38d3f99b14d6bb6005775a83c3ad7" member_type: VOTER last_known_addr { host: "127.0.222.61" port: 43725 } } peers { permanent_uuid: "5f6969f80d344471a170cf800dc581a6" member_type: VOTER last_known_addr { host: "127.0.222.60" port: 37181 } } }
I20260924 00:55:27.569248 24197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:55:27.693773 23997 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:27.689072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.0.222.62:46199)
0924 00:55:27.689343 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.689349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.691670 (+ 2321us) server_negotiation.cc:407] Connection header received
0924 00:55:27.692000 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.692007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.692100 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.692176 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.693472 (+ 1296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56974: BlockingRecv error: recv got EOF from 127.0.0.1:56974 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 00:55:27.696612 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:27.691397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.0.222.60:37181)
0924 00:55:27.691602 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.691609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.694406 (+ 2797us) server_negotiation.cc:407] Connection header received
0924 00:55:27.694721 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.694727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.694820 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.694897 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.696321 (+ 1424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54592: BlockingRecv error: recv got EOF from 127.0.0.1:54592 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 00:55:27.702466 24212 catalog_manager.cc:1259] Loaded cluster ID: 756be449629e4572b76e122e51c31ae3
I20260924 00:55:27.702596 24212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7: loading cluster ID for follower catalog manager: success
I20260924 00:55:27.706796 24212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7: acquiring CA information for follower catalog manager: success
W20260924 00:55:27.710616 24073 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:27.699040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.0.222.61:43725)
0924 00:55:27.700408 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.700415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.700443 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:55:27.700527 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.700534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.700654 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.700778 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.710355 (+ 9577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45998: BlockingRecv error: recv got EOF from 127.0.0.1:45998 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 00:55:27.713991 24212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ad38d3f99b14d6bb6005775a83c3ad7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:55:27.758528 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.742564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.0.222.61:43725)
0924 00:55:27.742771 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.742779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.749820 (+ 7041us) server_negotiation.cc:407] Connection header received
0924 00:55:27.750580 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.750588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.750817 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.750978 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:27.751780 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.752904 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.755114 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.755767 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.755870 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.756696 (+ 826us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:27.757497 (+ 801us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:27.757627 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.758082 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.758181 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 00:55:27.811875 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:27.798490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.0.222.60:37181)
0924 00:55:27.798713 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.798720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.809382 (+ 10662us) server_negotiation.cc:407] Connection header received
0924 00:55:27.810068 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.810075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.810261 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.810360 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.811591 (+ 1231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54602: BlockingRecv error: recv got EOF from 127.0.0.1:54602 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 00:55:27.870335 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.855990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.0.222.61:43725)
0924 00:55:27.856197 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.856204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.863331 (+ 7127us) server_negotiation.cc:407] Connection header received
0924 00:55:27.863891 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.863898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.864045 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.864116 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:27.864847 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.866003 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.868068 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.868765 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.868857 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.869501 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:27.869666 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:27.869727 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.869933 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.870024 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 00:55:27.917624 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:27.904537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.0.222.60:37181)
0924 00:55:27.904739 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.904745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.915477 (+ 10732us) server_negotiation.cc:407] Connection header received
0924 00:55:27.916064 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.916071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.916171 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.916242 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.917357 (+ 1115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54616: BlockingRecv error: recv got EOF from 127.0.0.1:54616 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:55:27.941514 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.935306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.0.222.61:43725)
0924 00:55:27.935539 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.935546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.936082 (+ 536us) server_negotiation.cc:407] Connection header received
0924 00:55:27.936283 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.936288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.936363 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.936449 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:27.936885 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.937823 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.939334 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.939807 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.939930 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.940651 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:27.940812 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:27.940878 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.941125 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.941240 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 00:55:27.951119 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:27.947930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.222.60:37181)
0924 00:55:27.948124 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.948130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.949733 (+ 1603us) server_negotiation.cc:407] Connection header received
0924 00:55:27.949899 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.949905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.950004 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.950093 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.950857 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54628: BlockingRecv error: recv got EOF from 127.0.0.1:54628 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 00:55:27.967835 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.961516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.0.222.61:43725)
0924 00:55:27.961746 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.961752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.962398 (+ 646us) server_negotiation.cc:407] Connection header received
0924 00:55:27.962701 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.962706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.962780 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.962856 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:27.963320 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.964068 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.965421 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.965914 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.966027 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:27.966458 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:27.966572 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:27.966653 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:27.967519 (+ 866us) server_negotiation.cc:299] Negotiation successful
0924 00:55:27.967610 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 00:55:27.979293 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:27.974351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.0.222.60:37181)
0924 00:55:27.974672 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.974678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.977850 (+ 3172us) server_negotiation.cc:407] Connection header received
0924 00:55:27.978041 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.978047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.978144 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.978234 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:27.979054 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54632: BlockingRecv error: recv got EOF from 127.0.0.1:54632 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
I20260924 00:55:28.000890 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:27.995220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.0.222.61:43725)
0924 00:55:27.995443 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:27.995449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:27.995992 (+ 543us) server_negotiation.cc:407] Connection header received
0924 00:55:27.996187 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:27.996192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:27.996272 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:27.996348 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:27.996788 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.997704 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.999140 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:27.999600 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:27.999712 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.000138 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.000267 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.000342 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.000491 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.000573 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 00:55:28.009234 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:28.007377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.222.60:37181)
0924 00:55:28.007557 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.007562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.007676 (+ 114us) server_negotiation.cc:407] Connection header received
0924 00:55:28.007960 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.007965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.008044 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.008139 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:28.008931 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54642: BlockingRecv error: recv got EOF from 127.0.0.1:54642 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:55:28.034808 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:28.028686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.0.222.61:43725)
0924 00:55:28.028884 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.028891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.029428 (+ 537us) server_negotiation.cc:407] Connection header received
0924 00:55:28.029633 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.029637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.029705 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.029787 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:28.030239 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.031271 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.032899 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.033469 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.033600 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.034078 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.034207 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.034288 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.034440 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.034528 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 00:55:28.050987 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:28.048684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.0.222.60:37181)
0924 00:55:28.048863 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.048868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.049591 (+ 723us) server_negotiation.cc:407] Connection header received
0924 00:55:28.049788 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.049792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.049871 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.049960 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:28.050746 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54652: BlockingRecv error: recv got EOF from 127.0.0.1:54652 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:55:28.093542 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:28.087074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.0.222.61:43725)
0924 00:55:28.087326 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.087333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.087716 (+ 383us) server_negotiation.cc:407] Connection header received
0924 00:55:28.087930 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.087936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.088050 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.088163 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:28.088653 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.089803 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.091363 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.092106 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.092252 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.092676 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.092819 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.092917 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.093067 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.093162 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 00:55:28.102020 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:28.099880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.0.222.60:37181)
0924 00:55:28.100099 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.100105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.100443 (+ 338us) server_negotiation.cc:407] Connection header received
0924 00:55:28.100668 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.100672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.100750 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.100837 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:28.101796 (+ 959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54662: BlockingRecv error: recv got EOF from 127.0.0.1:54662 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 00:55:28.176618 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:28.170324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.0.222.61:43725)
0924 00:55:28.170523 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.170530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.170991 (+ 461us) server_negotiation.cc:407] Connection header received
0924 00:55:28.171197 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.171203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.171293 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.171397 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:28.171926 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.173015 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.174566 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.175238 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.175360 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.175833 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.175965 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.176050 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.176231 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.176319 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 00:55:28.195858 24146 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:55:28.193947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.0.222.60:37181)
0924 00:55:28.194130 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.194136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.194499 (+ 363us) server_negotiation.cc:407] Connection header received
0924 00:55:28.194705 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.194711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.194817 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.194931 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:55:28.195597 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54668: BlockingRecv error: recv got EOF from 127.0.0.1:54668 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 00:55:28.296196 24160 catalog_manager.cc:1259] Loaded cluster ID: 756be449629e4572b76e122e51c31ae3
I20260924 00:55:28.296293 24160 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: loading cluster ID for follower catalog manager: success
I20260924 00:55:28.298995 24160 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: acquiring CA information for follower catalog manager: success
I20260924 00:55:28.300776 24160 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f6969f80d344471a170cf800dc581a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:55:28.332443 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:28.327369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.0.222.61:43725)
0924 00:55:28.327550 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.327554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.327846 (+ 292us) server_negotiation.cc:407] Connection header received
0924 00:55:28.328014 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.328018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.328098 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.328182 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:28.328572 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.329393 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.330883 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.331303 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.331402 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.331835 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.331960 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.332029 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.332172 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.332237 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 00:55:28.344698 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:28.338941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.0.222.60:37181)
0924 00:55:28.339156 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.339162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.339483 (+ 321us) server_negotiation.cc:407] Connection header received
0924 00:55:28.339684 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.339688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.339844 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.339946 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:28.340372 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.341138 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.342489 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.342917 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.343034 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.343552 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.344062 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.344207 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.344312 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.344409 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 00:55:28.357582 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:28.350575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.222.61:43725)
0924 00:55:28.351183 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.351190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.351222 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:55:28.351590 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.351598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.351694 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.351758 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:28.352643 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.353745 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.355510 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.356194 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.356314 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.356778 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.356885 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.356959 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.357183 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.357288 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431}
I20260924 00:55:28.359018 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:28.350925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.0.222.60:37181)
0924 00:55:28.351929 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.351935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.352596 (+ 661us) server_negotiation.cc:407] Connection header received
0924 00:55:28.352840 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.352844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.352911 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.352988 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:28.353588 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.354638 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.356521 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.357168 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.357296 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.358263 (+ 967us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.358391 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.358474 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.358665 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.358744 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880}
I20260924 00:55:28.359515 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:55:28.350665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.0.222.62:46199)
0924 00:55:28.351913 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0924 00:55:28.351918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:55:28.352779 (+ 861us) server_negotiation.cc:407] Connection header received
0924 00:55:28.353081 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:55:28.353087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:55:28.353320 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:55:28.353465 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:55:28.354075 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.355053 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.357092 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:55:28.357706 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:55:28.357818 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:55:28.358285 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:55:28.358630 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:55:28.358725 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 00:55:28.359152 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 00:55:28.359226 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":1043,"threads_started":1}
I20260924 00:55:28.363910 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 23928
I20260924 00:55:28.364981 23990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:28.474952 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 23928
I20260924 00:55:28.491899 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 24002
I20260924 00:55:28.493008 24064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:28.605021 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 24002
I20260924 00:55:28.621147 888 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfVll0B/build/asan/bin/kudu pid 24076
I20260924 00:55:28.622347 24138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 00:55:28.730660 888 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfVll0B/build/asan/bin/kudu with pid 24076
2026-09-24T00:55:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4299 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8546 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366022 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST